System and method for efficient database record replication using different replication strategies based on the database records

ABSTRACT

Systems and methods are provided for synchronizing operational data records in a manner that reduces redundant copying of data. A set of operational data records is stored, wherein each data record is assigned a unique ID in ascending order based on a creation time of the data record. A highest unique ID is identified from the operational data records in the set of operational data records. A tolerance number is identified that is indicative of a range of unique IDs that can be processed at a same time such that it cannot be guaranteed that operational data records with unique IDs separated by less than the tolerance number were assigned unique IDs in the order that the operational data records were created. A synchronization ID is calculated comprising subtracting the tolerance number from the highest unique ID. The synchronization ID is transmitted to the source operational data store.

CROSS-REFERENCES TO RELATED APPLICATIONS

The present application relates to and claims priority under 35 U.S.C.119(e) to U.S. Provisional Application No. 61/661,135, filed on Jun. 18,2012 and entitled “Enhanced Data Management Virtualization System,” thedisclosure of which is hereby incorporated by reference herein in itsentirety.

This application is also related to the following applications, filedherewith and hereby incorporated by reference:

-   -   “System and Method for Efficient Database Record Replication        Using Different Replication Strategies Based on the Database        Records” (U.S. application Ser. No. 13/920,932);    -   “System and Method for Caching Hashes for Co-Located Data in a        Deduplication Data Store” (U.S. application Ser. No.        13/920,923);    -   “System and Method for Incrementally Backing Up Out-of-band        Data” (U.S. application Ser. No. 13/920,981);    -   “System and Method for Providing Intra-Process Communication for        an Application Programming Interface” (U.S. application Ser. No.        13/920,976); and    -   “System and Method for Quick-Linking User Interface Jobs Across        Services Based on System Implementation Information” (U.S.        application Ser. No. 13/921,008); and    -   “System and Method for Intelligent Database Backup” (U.S.        application Ser. No. 13/920,950).

BACKGROUND

The business requirements for managing the lifecycle of application datahave been traditionally met by deploying multiple point solutions, eachof which addresses a part of the lifecycle. This has resulted in acomplex and expensive infrastructure where multiple copies of data arecreated and moved multiple times to individual storage repositories. Theadoption of server virtualization has become a catalyst for simple,agile and low-cost compute infrastructure. This has led to largerdeployments of virtual hosts and storage, further exacerbating the gapbetween the emerging compute models and the current data managementimplementations.

Applications that provide business services depend on storage of theirdata at various stages of its lifecycle. FIG. 1 shows a typical set ofdata management operations that would be applied to the data of anapplication such as a database underlying a business service such aspayroll management. In order to provide a business service, application102 requires primary data storage 122 with some contracted level ofreliability and availability.

Backups 104 are made to guard against corruption or the primary datastorage through hardware or software failure or human error. Typicallybackups may be made daily or weekly to local disk or tape 124, and movedless frequently (weekly or monthly) to a remote physically securelocation 125.

Concurrent development and test 106 of new applications based on thesame database requires a development team to have access to another copyof the data 126. Such a snapshot might be made weekly, depending ondevelopment schedules.

Compliance with legal or voluntary policies 108 may require that somedata be retained for safely future access for some number of years;usually data is copied regularly (say, monthly) to a long-term archivingsystem 128.

Disaster Recovery services 110 guard against catastrophic loss of dataif systems providing primary business services fail due to some physicaldisaster. Primary data is copied 130 to a physically distinct locationas frequently as is feasible given other constraints (such as cost). Inthe event of a disaster the primary site can be reconstructed and datamoved back from the safe copy.

Business Continuity services 112 provide a facility for ensuringcontinued business services should the primary site become compromised.Usually this requires a hot copy 132 of the primary data that is innear-lockstep with the primary data, as well as duplicate systems andapplications and mechanisms for switching incoming requests to theBusiness Continuity servers.

Thus, data management is currently a collection of point applicationsmanaging the different parts of the lifecycle. This has been an artifactof evolution of data management solutions over the last two decades.

SUMMARY OF THE INVENTION

In accordance with the disclosed subject matter, systems, methods, andnon-transitory computer-readable media are provided for efficientdatabase record replication using different replication strategies basedon the database records.

The disclosed subject matter includes a method for synchronizingoperational data records generated during data management operations ina manner that reduces redundant copying of data during synchronization.The method includes storing, by a central operational data store, a setof operational data records, wherein each data record is assigned aunique identifier (ID) in ascending order based on a creation time ofthe data record. The method includes identifying, by the centraloperational data store, a highest unique ID from the operational datarecords in the set of operational data records. The method includesidentifying, by the central operational data store, a tolerance numberthat is indicative of a range of unique IDs that can be processed at asame time such that it cannot be guaranteed that operational datarecords with unique IDs separated by less than the tolerance number wereassigned unique IDs in the order that the operational data records werecreated. The method includes calculating, by the central operationaldata store, a synchronization ID including subtracting the tolerancenumber from the highest unique ID. The method includes transmitting, bythe central operational data store, the synchronization ID to the sourceoperational data store to instruct the source operational data store totransmit any operational data records stored at the source operationaldata store with unique IDs greater than the synchronization ID.

In some embodiments, a set of unique IDs is identified from the set ofoperational data records, wherein each unique ID in the set of uniqueIDs is between the highest unique ID and the synchronization ID, and theset of unique IDs is transmitted to the source operational data store toinstruct the source operational data store to not transmit operationaldata records with the set of unique IDs. The highest unique ID can betransmitted to the source operational data store to instruct the sourceoperational data store to not transmit operational data records with thehighest unique ID.

In some embodiments, the tolerance number is calculated based on anumber of concurrent transactions allowed and a maximum delay. Thenumber of concurrent transactions can be doubled. Each unique ID can bea monotonically-increasing unique ID. The set of operational datarecords can be a large set of operational data records, including over100,000 operational data records, and the data records in the set ofoperational data records are not changing once created.

The disclosed subject matter includes a method for synchronizingoperational data records generated during data management operations ina manner that reduces redundant copying of data during synchronization.The method includes storing, by a central operational data store, a setof operational data records, wherein each data record is assigned atimestamp based on either a creation time, or modification time of thedata record. The method includes identifying, by the central operationaldata store, a last backup time of the set of operational data recordsfrom a source operational data store. The method includes calculating,by the central operational data store, a tolerance number based on anelapsed time that is indicative of a range of timestamps that can beprocessed at a same time such that it cannot be guaranteed thatoperational data records with timestamps separated by less than thetolerance number were assigned timestamps in the order that theoperational data records were created, modified, or both. The methodincludes calculating, by the central operational data store, asynchronization timestamp including subtracting the tolerance numberfrom the last backup time. The method includes transmitting, by thecentral operational data store, the synchronization timestamp to thesource operational data store to instruct the source operational datastore to transmit any operational data records stored at the sourceoperational data store with timestamps greater than the synchronizationtimestamp.

In some embodiments, a reply is received including a set of operationaldata records, each with a timestamp occurring after the synchronizationtimestamp, and a list of unique IDs for each data record stored by thesource operational data store. Any operational data records in the setof operational data records with a unique ID that is not in the list ofunique IDs can be deleted. The set of operational data records can be amedium set of operational data records, including more than 1,000operational data records but less than 100,000 operational data records.

These and other capabilities of the disclosed subject matter will bemore fully understood after a review of the following figures, detaileddescription, and claims. It is to be understood that the phraseology andterminology employed herein are for the purpose of description andshould not be regarded as limiting.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified diagram of current methods deployed to manage thedata lifecycle for a business service.

FIG. 2 is an overview of the management of data throughout its lifecycleby a single Data Management Virtualization System.

FIG. 3 is a simplified block diagram of the Data ManagementVirtualization system.

FIG. 4 is a view of the Data Management Virtualization Engine.

FIG. 5 illustrates the Object Management and Data Movement Engine.

FIG. 6 shows the Storage Pool Manager.

FIG. 7 shows the decomposition of the Service Level Agreement.

FIG. 8 illustrates the Application Specific Module.

FIG. 9 shows the Service Policy Manager.

FIG. 10 is a flowchart of the Service Policy Scheduler.

FIG. 11 is a block diagram of the Content Addressable Storage (CAS)provider.

FIG. 12 shows the definition of an object handle within the CAS system.

FIG. 13 shows the data model and operations for the temporalrelationship graph stored for objects within the CAS.

FIG. 14 is a diagram representing the operation of a garbage collectionalgorithm in the CAS.

FIG. 15 is a flowchart for the operation of copying an object into theCAS.

FIG. 16 is a system diagram of a typical deployment of the DataManagement Virtualization system.

FIG. 17 is a schematic diagram of the VSS framework on a MicrosoftWindows operating system in the prior art.

FIG. 18A illustrates a combined VSS requestor and VSS provider, inaccordance with some embodiments.

FIG. 18B illustrates an exemplary data flow for a combined VSS requestorand VSS provider, in accordance with some embodiments.

FIG. 19A is a schematic diagram of a system providing out-of-bandprotection, in accordance with some embodiments.

FIG. 19B is an exemplary computerized method for providing out-of-bandprotection, in accordance with some embodiments.

FIG. 20A is a schematic diagram for protection of a database inaccordance with some embodiments.

FIG. 20B depicts a computerized method for protection of a database inaccordance with some embodiments.

FIG. 20C depicts a computerized method for protection of a database inaccordance with some embodiments.

FIG. 21 depicts a Desktop, which is the user interface that implementssystems and methods for Protection and Availability (PAS) storageappliance, in accordance with some embodiments.

FIG. 22 depicts a schematic diagram of a service manager interface withquick links, in accordance with some embodiments.

FIG. 22A depicts a schematic diagram of a service object, in accordancewith some embodiments.

FIG. 23 depicts an Application Manager Service, in accordance with someembodiments.

FIG. 24 depicts a System Monitor service, which in the Desktop isresponsible for handling all user visible activities related to jobs,including monitoring and management, in accordance with someembodiments.

FIG. 25 depicts the display of specific details of a particular job fromwithin a System Monitor service, in accordance with some embodiments.

FIG. 26 describes the user flow without Quick-linking, in accordancewith some embodiments.

FIG. 27 describes the user flow with Quick-linking, in accordance withsome embodiments.

FIG. 28 depicts a Management Console, in accordance with someembodiments.

FIG. 29 depicts Enterprise Manager Operational Data, in accordance withsome embodiments.

FIG. 30A depicts examples of data management operational data, inaccordance with some embodiments.

FIG. 30B depicts examples of protection data for medium-sized data indata management operational data, in accordance with some embodiments.

FIG. 30C depicts an example of a synchronization request formedium-sized data in data management operational data, in accordancewith some embodiments.

FIG. 30D depicts an example of a response to the synchronization requestfor medium-sized data in data management operational data, in accordancewith some embodiments.

FIG. 30E depicts an example of history data for large-sized data in datamanagement operational data, in accordance with some embodiments.

FIG. 30F depicts an example of a synchronization request for historydata for large-sized data in data management operational data, inaccordance with some embodiments.

FIG. 30G depicts an example of a response to the synchronization requestfor history data for large-sized data in data management operationaldata, in accordance with some embodiments.

FIG. 31 is a schematic diagram of a persist header in accordance withsome embodiments.

FIG. 32 is a schematic diagram of a hash index in accordance with someembodiments.

FIG. 33 is a schematic diagram of index pages in a storage system inaccordance with some embodiments.

FIG. 34 is a schematic diagram of a page cache in accordance with someembodiments.

FIG. 35 is a schematic diagram of a key/value hash table in accordancewith some embodiments.

FIG. 36 is a flowchart depicting the operation of a system that uses ascoreboard to find a hash, in accordance with some embodiments.

FIG. 37 is a flowchart depicting the operation of a scoreboard whenfinding a hash in accordance with some embodiments.

FIG. 38 is a diagram that depicts the various components of acomputerized system upon which certain elements may be implemented,according to certain embodiments.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forthregarding the systems and methods of the disclosed subject matter andthe environment in which such systems and methods may operate, etc., inorder to provide a thorough understanding of the disclosed subjectmatter. It will be apparent to one skilled in the art, however, that thedisclosed subject matter may be practiced without such specific details,and that certain features, which are well known in the art, are notdescribed in detail in order to avoid unnecessary complication of thedisclosed subject matter. In addition, it will be understood that theembodiments provided below are exemplary, and that it is contemplatedthat there are other systems and methods that are within the scope ofthe disclosed subject matter.

Current Data Management architecture and implementations such asdescribed above involve multiple applications addressing different partsof data lifecycle management, all of them performing certain commonfunctions: (a) make a copy of application data (the frequency of thisaction is commonly termed the Recovery Point Objective (RPO)), (b) storethe copy of data in an exclusive storage repository, typically in aproprietary format, and (c) retain the copy for certain duration,measured as Retention Time. A primary difference in each of the pointsolutions is in the frequency of the RPO, the Retention Time, and thecharacteristics of the individual storage repositories used, includingcapacity, cost and geographic location.

This disclosure pertains to Data Management Virtualization. DataManagement activities, such as Backup, Replication and Archiving arevirtualized in that they do not have to be configured and runindividually and separately. Instead, the user defines their businessrequirement with regard to the lifecycle of the data, and the DataManagement Virtualization System performs these operationsautomatically. A snapshot is taken from primary storage to secondarystorage; this snapshot is then used for a backup operation to othersecondary storage. Essentially an arbitrary number of these backups maybe made, providing a level of data protection specified by a ServiceLevel Agreement.

This disclosure also pertains to a method of storing deduplicated imagesin which a portion of the image is stored in encoded form directly in ahash table, the method including organizing unique content of each dataobject as a plurality of content segments and storing the contentsegments in a data store; for each data object, creating an organizedarrangement of hash structures, wherein each structure, for a subset ofthe hash structures, includes a field to contain a hash signature for acorresponding content segment and is associated with a reference to thecorresponding content segment, wherein the logical organization of thearrangement represents the logical organization of the content segmentsas they are represented within the data object; receiving content to beincluded in the deduplicated image of the data object; determining ifthe received content may be encoded using a predefined non-lossyencoding technique and in which the encoded value would fit within thefield for containing a hash signature; if so, placing the encoding inthe field and marking the hash structure to indicate that the fieldcontains encoded content for the deduplicated image; if not, generatinga hash signature for the received content and placing the hash signaturein the field and placing the received content in a corresponding contentsegment in said data store if it is unique.

Data Management Virtualization technology according to this disclosureis based on an architecture and implementation based on the followingguiding principles.

First, define the business requirements of an application with a ServiceLevel Agreement (SLA) for its entire data lifecycle. The SLA is muchmore than a single RPO, Retention and Recovery Time Objective (RTO). Itdescribes the data protection characteristics for each stage of the datalifecycle. Each application may have a different SLA.

Second, provide a unified Data Management Virtualization Engine thatmanages the data protection lifecycle, moving data across the variousstorage repositories, with improved storage capacity and networkbandwidth. The Data Management Virtualization system achieves theseimprovements by leveraging extended capabilities of modern storagesystems by tracking the portions of the data that have changed over timeand by data deduplication and compression algorithms that reduce theamount of data that needs to be copied and moved.

Third, leverage a single master copy of the application data to be thebasis for multiple elements within the lifecycle. Many of the DataManagement operations such as backup, archival and replication depend ona stable, consistent copy of the data to be protected. The DataManagement Virtualization System leverages a single copy of the data formultiple purposes. A single instance of the data maintained by thesystem may serve as the source, from which each data management functionmay make additional copies as needed. This contrasts with requiringapplication data to be copied multiple times by multiple independentdata management applications in the traditional approach.

Fourth, abstracting physical storage resources into a series of dataprotection storage pools, which are virtualized out of different classesof storage including local and remote disk, solid state memory, tape andoptical media, private, public and/or hybrid storage clouds. The storagepools provide access independent of the type, physical location orunderlying storage technology. Business requirements for the lifecycleof data may call for copying the data to different types of storagemedia at different times. The Data Management Virtualization systemallows the user to classify and aggregate different storage media intostorage pools, for example, a Quick Recovery Pool, which consists ofhigh speed disks, and a Cost Efficient Long-term Storage Pool, which maybe a deduplicated store on high capacity disks, or a tape library. TheData Management Virtualization System can move data amongst these poolsto take advantage of the unique characteristics of each storage medium.The abstraction of Storage Pools provides access independent of thetype, physical location or underlying storage technology.

Fifth, improve the movement of the data between storage pools anddisaster locations utilizing underlying device capabilities andpost-deduplicated application data. The Data Management VirtualizationSystem discovers the capabilities of the storage systems that comprisethe Storage Pools, and takes advantage of these capabilities to movedata efficiently. If the Storage System is a disk array that supportsthe capability of creating a snapshot or clone of a data volume, theData Management Virtualization System will take advantage of thiscapability and use a snapshot to make a copy of the data rather thanreading the data from one place and writing it to another. Similarly, ifa storage system supports change tracking, the Data ManagementVirtualization System will update an older copy with just the changes toefficiently create a new copy. When moving data across a network, theData Management Virtualization system uses a deduplication andcompression algorithm that avoids sending data that is already availableon the other side of the network.

One key aspect of improving data movement is recognizing thatapplication data changes slowly over time. A copy of an application thatis made today will, in general, have a lot of similarities to the copyof the same application that was made yesterday. In fact today's copy ofthe data could be represented as yesterday's copy with a series of deltatransformations, where the size of the delta transformations themselvesare usually much smaller than all of the data in the copy itself. TheData Management Virtualization system captures and records thesetransformations in the form of bitmaps or extent lists. In oneembodiment of the system, the underlying storage resources—a disk arrayor server virtualization system—are capable of tracking the changes madeto a volume or file; in these environments, the Data ManagementVirtualization system queries the storage resources to obtain thesechange lists, and saves them with the data being protected.

In the preferred embodiment of the Data Management Virtualizationsystem, there is a mechanism for eavesdropping on the primary dataaccess path of the application, which enables the Data ManagementVirtualization system to observe which parts of the application data aremodified, and to generate its own bitmap of modified data. If, forexample, the application modifies blocks 100, 200 and 300 during aparticular period, the Data Management Virtualization system willeavesdrop on these events, and create a bitmap that indicates that theseparticular blocks were modified. When processing the next copy ofapplication data, the Data Management Virtualization system will onlyprocess blocks 100, 200 and 300 since it knows that these were the onlyblocks that were modified.

In one embodiment of the system, where the primary storage for theapplication is a modern disk array or storage virtualization appliance,the Data Management Virtualization system takes advantage of apoint-in-time snapshot capability of an underlying storage device tomake the initial copy of the data. This virtual copy mechanism is afast, efficient and low-impact technique of creating the initial copythat does not guarantee that all the bits will be copied, or storedtogether. Instead, virtual copies are constructed by maintainingmetadata and data structures, such as copy-on-write volume bitmaps orextents, that allow the copies to be reconstructed at access time. Thecopy has a lightweight impact on the application and on the primarystorage device. In another embodiment, where the application is based ona Server Virtualization System such as VMware or Xen, the DataManagement Virtualization system uses the similarvirtual-machine-snapshot capability that is built into the ServerVirtualization systems. When a virtual copy capability is not available,the Data Management Virtualization System may include its own built-insnapshot mechanism.

It is possible to use the snapshot as a data primitive underlying all ofthe data management functions supported by the system. Because it islightweight, the snapshot can be used as an internal operation even whenthe requested operation is not a snapshot per se; it is created toenable and facilitate other operations.

At the time of creation of a snapshot, there may be certain preparatoryoperations involved in order to create a coherent snapshot or coherentimage, such that the image may be restored to a state that is usable bythe application. These preparatory operations need only be performedonce, even if the snapshot will be leveraged across multiple datamanagement functions in the system, such as backup copies which arescheduled according to a policy. The preparatory operations may includeapplication quiescence, which includes flushing data caches and freezingthe state of the application; it may also include other operations knownin the art and other operations useful for retaining a complete image,such as collecting metadata information from the application to bestored with the image.

FIG. 2 illustrates one way that a Virtualized Data Management system canaddress the data lifecycle requirements described earlier in accordancewith these principles.

To serve local backup requirements, a sequence of efficient snapshotsare made within local high-availability storage 202. Some of thesesnapshots are used to serve development/test requirements without makinganother copy. For longer term retention of local backup, a copy is madeefficiently into long-term local storage 204, which in thisimplementation uses deduplication to reduce repeated copying. The copieswithin long-term storage may be accessed as backups or treated as anarchive, depending on the retention policy applied by the SLA. A copy ofthe data is made to remote storage 206 in order to satisfy requirementsfor remote backup and business continuity—again a single set of copiessuffices both purposes. As an alternative for remote backup and disasterrecovery, a further copy of the data may be made efficiently to arepository 208 hosted by a commercial or private cloud storage provider.

The Data Management Virtualization System

FIG. 3 illustrates the high level components of the Data ManagementVirtualization System that implements the above principles. Preferably,the system comprises these basic functional components further describedbelow.

Application 300 creates and owns the data. This is the software systemthat has been deployed by the user, as for example, an email system, adatabase system, or financial reporting system, in order to satisfy somecomputational need. The Application typically runs on a server andutilizes storage. For illustrative purposes, only one application hasbeen indicated. In reality there may be hundreds or even thousands ofapplications that are managed by a single Data Management VirtualizationSystem.

Storage Resources 302 is where application data is stored through itslifecycle. The Storage Resources are the physical storage assets,including internal disk drives, disk arrays, optical and tape storagelibraries and cloud-based storage systems that the user has acquired toaddress data storage requirements. The storage resources consist ofPrimary Storage 310, where the online, active copy of the applicationdata is stored, and Secondary Storage 312 where additional copies of theapplication data are stored for the purposes such as backup, disasterrecovery, archiving, indexing, reporting and other uses. Secondarystorage resources may include additional storage within the sameenclosure as the primary storage, as well as storage based on similar ordifferent storage technologies within the same data center, anotherlocation or across the internet.

One or more Management Workstations 308 allow the user to specify aService Level Agreement (SLA) 304 that defines the lifecycle for theapplication data. A Management workstation is a desktop or laptopcomputer or a mobile computing device that is used to configure, monitorand control the Data Management Virtualization System. A Service LevelAgreement is a detailed specification that captures the detailedbusiness requirements related to the creation, retention and deletion ofsecondary copies of the application data. The SLA is much more than thesimple RTO and RPO that are used in traditional data managementapplications to represent the frequency of copies and the anticipatedrestore time for a single class of secondary storage. The SLA capturesthe multiple stages in the data lifecycle specification, and allows fornon uniform frequency and retention specifications within each class ofsecondary storage. The SLA is described in greater detail in FIG. 7.

Data Management Virtualization Engine 306 manages all of the lifecycleof the application data as specified in SLA. It manages potentially alarge number of SLAs for a large number of applications. The DataManagement Virtualization Engine takes inputs from the user through theManagement Workstation and interacts with the applications to discoverthe applications primary storage resources. The Data ManagementVirtualization Engine makes decisions regarding what data needs to beprotected and what secondary storage resources best fulfill theprotection needs. For example, if an enterprise designates itsaccounting data as requiring copies to be made at very short intervalsfor business continuity purposes as well as for backup purposes, theEngine may decide to create copies of the accounting data at a shortinterval to a first storage pool, and to also create backup copies ofthe accounting data to a second storage pool at a longer interval,according to an appropriate set of SLAs. This is determined by thebusiness requirements of the storage application.

The Engine then makes copies of application data using advancedcapabilities of the storage resources as available. In the aboveexample, the Engine may schedule the short-interval business continuitycopy using a storage appliance's built-in virtual copy or snapshotcapabilities. Data Management Virtualization Engine moves theapplication data amongst the storage resources in order to satisfy thebusiness requirements that are captured in the SLA. The Data ManagementVirtualization Engine is described in greater detail in FIG. 4.

The Data Management Virtualization System as a whole may be deployedwithin a single host computer system or appliance, or it may be onelogical entity but physically distributed across a network ofgeneral-purpose and purpose-built systems. Certain components of thesystem may also be deployed within a computing or storage cloud.

In one embodiment of the Data Management Virtualization System the DataManagement Virtualization Engine largely runs as multiple processes on afault tolerant, redundant pair of computers. Certain components of theData Management Virtualization Engine may run close to the applicationwithin the application servers. Some other components may run close tothe primary and secondary storage, within the storage fabric or in thestorage systems themselves. The Management stations are typicallydesktop and laptop computers and mobile devices that connect over asecure network to the Engine.

The Data Management Virtualization Engine

FIG. 4 illustrates an architectural overview of the Data ManagementVirtualization Engine 306 according to certain embodiments of theinvention. The 306 Engine includes the following modules:

Application Specific Module 402: This module is responsible forcontrolling and collecting metadata from the application 300.Application metadata includes information about the application such asthe type of application, details about its configuration, location ofits datastores, its current operating state. Controlling the operationof the application includes actions such as flushing cached data todisk, freezing and thawing application I/O, rotating or truncating logfiles, and shutting down and restarting applications. The ApplicationSpecific module performs these operations and sends and receivesmetadata in responses to commands from the Service Level Policy Engine406, described below. The Application Specific Module is described inmore detail in connection with FIG. 8.

Service Level Policy Engine 406 acts on the SLA 304 provided by the userto make decisions regarding the creation, movement and deletion ofcopies of the application data. Each SLA describes the businessrequirements related to protection of one application. The Service LevelPolicy Engine analyzes each SLA and arrives at a series of actions eachof which involve the copying of application data from one storagelocation to another. The Service Level Policy Engine then reviews theseactions to determine priorities and dependencies, and schedules andinitiates the data movement jobs. The Service Level Policy Engine isdescribed in more detail in connection with FIG. 9.

Object Manager and Data Movement Engine 410 creates a composite objectconsisting of the Application data, the Application Metadata and the SLAwhich it moves through different storage pools per instruction from thePolicy Engine. The Object Manager receives instructions from the ServicePolicy Engine 406 in the form of a command to create a copy ofapplication data in a particular pool based on the live primary data 413belonging to the application 300, or from an existing copy, e.g., 415,in another pool. The copy of the composite object that is created by theObject Manager and the Data Movement Engine is self contained and selfdescribing in that it contains not only application data, but alsoapplication metadata and the SLA for the application. The Object Managerand Data Movement Engine are described in more detail in connection withFIG. 5.

Storage Pool Manager 412 is a component that adapts and abstracts theunderlying physical storage resources 302 and presents them as virtualstorage pools 418. The physical storage resources are the actual storageassets, such as disk arrays and tape libraries that the user hasdeployed for the purpose of supporting the lifecycle of the data of theuser's applications. These storage resources might be based on differentstorage technologies such as disk, tape, flash memory or opticalstorage. The storage resources may also have different geographiclocations, cost and speed attributes, and may support differentprotocols. The role of the Storage Pool Manager is to combine andaggregate the storage resources, and mask the differences between theirprogramming interfaces. The Storage Pool Manager presents the physicalstorage resources to the Object Manager 410 as a set of storage poolsthat have characteristics that make these pools suitable for particularstages in the lifecycle of application data. The Storage Pool Manager isdescribed in more detail in connection with FIG. 6.

Object Manager and Data Movement Engine

FIG. 5 illustrates the Object Manager and Data Movement Engine 410. TheObject Manager and Data Movement Engine discovers and uses VirtualStorage Resources 510 presented to it by the Pool Managers 504. Itaccepts requests from the Service Level Policy Engine 406 to create andmaintain Data Storage Object instances from the resources in a VirtualStorage Pool, and it copies application data among instances of storageobjects from the Virtual Storage Pools according to the instructionsfrom the Service Level Policy Engine. The target pool selected for thecopy implicitly designates the business operation being selected, e.g.backup, replication or restore. The Service Level Policy Engine resideseither locally to the Object Manager (on the same system) or remotely,and communicates using a protocol over standard networkingcommunication. TCP/IP may be used in a preferred embodiment, as it iswell understood, widely available, and allows the Service Level PolicyEngine to be located locally to the Object Manager or remotely withlittle modification.

In one embodiment, the system may deploy the Service Level Policy Engineon the same computer system as the Object Manager for ease ofimplementation. In another embodiment, the system may employ multiplesystems, each hosting a subset of the components if beneficial orconvenient for an application, without changing the design.

The Object Manager 501 and the Storage Pool Managers 504 are softwarecomponents that may reside on the computer system platform thatinterconnects the storage resources and the computer systems that usethose storage resources, where the user's application resides. Theplacement of these software components on the interconnect platform isdesignated as a preferred embodiment, and may provide the ability toconnect customer systems to storage via communication protocols widelyused for such applications (e.g. Fibre Channel, iSCSI, etc.), and mayalso provide ease of deployment of the various software components.

The Object Manager 501 and Storage Pool Manager 504 communicate with theunderlying storage virtualization platform via the ApplicationProgramming Interfaces made available by the platform. These interfacesallow the software components to query and control the behavior of thecomputer system and how it interconnects the storage resources and thecomputer system where the user's Application resides. The componentsapply modularity techniques as is common within the practice to allowreplacement of the intercommunication code particular to a givenplatform.

The Object Manager and Storage Pool Managers communicate via a protocol.These are transmitted over standard networking protocols, e.g. TCP/IP,or standard Interprocess Communication (IPC) mechanisms typicallyavailable on the computer system. This allows comparable communicationbetween the components if they reside on the same computer platform oron multiple computer platforms connected by a network, depending on theparticular computer platform. The current configuration has all of thelocal software components residing on the same computer system for easeof deployment. This is not a strict requirement of the design, asdescribed above, and can be reconfigured in the future as needed.

Object Manager

Object Manager 501 is a software component for maintaining Data StorageObjects, and provides a set of protocol operations to control it. Theoperations include creation, destruction, duplication, and copying ofdata among the objects, maintaining access to objects, and in particularallow the specification of the storage pool used to create copies. Thereis no common subset of functions supported by all pools; however, in apreferred embodiment, primary pools may be performance-optimized, i.e.lower latency, whereas backup or replication pools may becapacity-optimized, supporting larger quantities of data andcontent-addressable. The pools may be remote or local. The storage poolsare classified according to various criteria, including means by which auser may make a business decision, e.g. cost per gigabyte of storage.

First, the particular storage device from which the storage is drawn maybe a consideration, as equipment is allocated for different businesspurposes, along with associated cost and other practical considerations.Some devices may not even be actual hardware but capacity provided as aservice, and selection of such a resource can be done for practicalbusiness purposes.

Second, the network topological “proximity” is considered, as nearstorage is typically connected by low-latency, inexpensive networkresources, while distant storage may be connected by high-latency,bandwidth limited expensive network resources; conversely, the distanceof a storage pool relative to the source may be beneficial whengeographic diversity protects against a physical disaster affectinglocal resources.

Third, storage optimization characteristics are considered, where somestorage is optimized for space-efficient storage, but requirescomputation time and resources to analyze or transform the data beforeit can be stored, while other storage by comparison is “performanceoptimized,” taking more storage resources by comparison but usingcomparatively little computation time or resource to transform the data,if at all.

Fourth, “speed of access” characteristics are considered, where someresources intrinsic to a storage computer platform are readily andquickly made available to the user's Application, e.g. as a virtual SCSIblock device, while some can only be indirectly used. These ease andspeed of recovery is often governed by the kind of storage used, andthis allows it to be suitably classified.

Fifth, the amount of storage used and the amount available in a givenpool are considered, as there may be benefit to either concentrating orspreading the storage capacity used.

The Service Level Policy Engine, described below, combines the SLAprovided by the user with the classification criteria to determine howand when to maintain the application data, and from which storage poolsto draw the needed resources to meet the Service Level Agreement (SLA).

The object manager 501 creates, maintains and employs a historymechanism to track the series of operations performed on a data objectwithin the performance pools, and to correlate those operations withothers that move the object to other storage pools, in particularcapacity-optimized ones. This series of records for each data object ismaintained at the object manager for all data objects in the primarypool, initially correlated by primary data object, then correlated byoperation order: a time line for each object and a list of all such timelines. Each operation performed exploits underlying virtualizationprimitives to capture the state of the data object at a given point intime.

Additionally, the underlying storage virtualization appliance may bemodified to expose and allow retrieval of internal data structures, suchas bitmaps, that indicate the modification of portions of the datawithin the data object. These data structures are exploited to capturethe state of a data object at a point in time: e.g., a snapshot of thedata object, and to provide differences between snapshots taken at aspecific time, and thereby enables optimal backup and restore. While theparticular implementations and data structures may vary among differentappliances from different vendors, a data structure is employed to trackchanges to the data object, and storage is employed to retain theoriginal state of those portions of the object that have changed:indications in the data structure correspond to data retained in thestorage. When accessing the snapshot, the data structure is consultedand for portions that have been changed, the preserved data is accessedrather than the current data, as the data object has been modified atthe areas so indicated. A typical data structure employed is a bitmap,where each bit corresponds to a section of the data object. Setting thebit indicates that section has been modified after the point in time ofthe snapshot operation. The underlying snapshot primitive mechanismmaintains this for as long as the snapshot object exists.

The time line described above maintains a list of the snapshotoperations against a given primary data object, including the time anoperation is started, the time it is stopped (if at all), a reference tothe snapshot object, and a reference to the internal data structure(e.g. bitmaps or extent lists), so that it can be obtained from theunderlying system. Also maintained is a reference to the result ofcopying the state of the data object at any given point in time intoanother pool—as an example, copying the state of a data object into acapacity-optimized pool 407 using content addressing results in anobject handle. That object handle corresponds to a given snapshot and isstored with the snapshot operation in the time line. This correlation isused to identify suitable starting points.

Optimal backup and restore consult the list of operations from a desiredstarting point to an end point. A time ordered list of operations andtheir corresponding data structures (bitmaps) are constructed such thata continuous time series from start to finish is realized: there is nogap between start times of the operations in the series. This ensuresthat all changes to the data object are represented by the correspondingbitmap data structures. It is not necessary to retrieve all operationsfrom start to finish; simultaneously existing data objects andunderlying snapshots overlap in time; it is only necessary that thereare no gaps in time where a change might have occurred that was nottracked. As bitmaps indicate that a certain block of storage has changedbut not what the change is, the bitmaps may be added or composedtogether to realize a set of all changes that occurred in the timeinterval. Instead of using this data structure to access the state at apoint in time, the system instead exploits the fact that the datastructure represents data modified as time marches forward. Rather, theend state of the data object is accessed at the indicated areas, thusreturning the set of changes to the given data object from the givenstart time to the end time.

The backup operation exploits this time line, the correlated references,and access to the internal data structures to realize our backupoperation. Similarly, it uses the system in a complementary fashion toaccomplish our restore operation. The specific steps are described belowin the section for “Optimal Backup/Restore.”

Virtual Storage Pool Types

FIG. 5 illustrates several representative storage pool types. Althoughone primary storage pool and two secondary storage pools are depicted inthe figure, many more may be configured in some embodiments.

Primary Storage Pool 507—contains the storage resources used to createthe data objects in which the user Application stores its data. This isin contrast to the other storage pools, which exist to primarily fulfillthe operation of the Data Management Virtualization Engine.

Performance Optimized Pool 508—a virtual storage pool able to providehigh performance backup (i.e. point in time duplication, describedbelow) as well as rapid access to the backup image by the userApplication

Capacity Optimized Pool 509—a virtual storage pool that chiefly providesstorage of a data object in a highly space-efficient manner by use ofdeduplication techniques described below. The virtual storage poolprovides access to the copy of the data object, but does not do so withhigh performance as its chief aim, in contrast to the PerformanceOptimized pool above.

The initial deployments contain storage pools as described above, as aminimal operational set. The design fully expects multiple Pools of avariety of types, representing various combinations of the criteriaillustrated above, and multiple Pool Managers as is convenient torepresent all of the storage in future deployments. The tradeoffsillustrated above are typical of computer data storage systems.

From a practical point of view, these three pools represent a preferredembodiment, addressing most users requirements in a very simple way.Most users will find that if they have one pool of storage for urgentrestore needs, which affords quick recovery, and one other pool that islow cost, so that a large number of images can be retained for a largeperiod of time, almost all of the business requirements for dataprotection can be met with little compromise.

The format of data in each pool is dictated by the objectives andtechnology used within the pool. For example, the quick recovery pool ismaintained in the form very similar to the original data to minimize thetranslation required and to improve the speed of recovery. The long-termstorage pool, on the other hand, uses deduplication and compression toreduce the size of the data and thus reduce the cost of storage.

Object Management Operations 505

The Object Manager 501 creates and maintains instances of Data StorageObjects 503 from the Virtual Storage Pools 418 according to theinstructions sent to it by the Service Level Policy Engine 406. TheObject Manager provides data object operations in five major areas:point-in-time duplication or copying (commonly referred to as“snapshots”), standard copying, object maintenance, mapping and accessmaintenance, and collections.

Object Management operations also include a series of Resource Discoveryoperations for maintaining Virtual Storage Pools themselves andretrieving information about them. The Pool Manager 504 ultimatelysupplies the functionality for these.

Point-In-Time Copy (“Snapshot”) Operations

Snapshot operations create a data object instance representing aninitial object instance at a specific point in time. More specifically,a snapshot operation creates a complete virtual copy of the members of acollection using the resources of a specified Virtual Storage Pool. Thisis called a Data Storage Object. Multiple states of a Data StorageObject are maintained over time, such that the state of a Data StorageObject as it existed at a point in time is available. As describedabove, a virtual copy is a copy implemented using an underlying storagevirtualization API that allows a copy to be created in a lightweightfashion, using copy-on-write or other in-band technologies instead ofcopying and storing all bits of duplicate data to disk. This may beimplemented using software modules written to access the capabilities ofan off-the-shelf underlying storage virtualization system such asprovided by EMC, vmware or IBM in some embodiments. Where suchunderlying virtualizations are not available, the described system mayprovide its own virtualization layer for interfacing with unintelligenthardware.

Snapshot operations require the application to freeze the state of thedata to a specific point so that the image data is coherent, and so thatthe snapshot may later be used to restore the state of the applicationat the time of the snapshot. Other preparatory steps may also berequired. These are handled by the Application-Specific Module 302,which is described in a subsequent section. For live applications,therefore, the most lightweight operations are desired.

Snapshot operations are used as the data primitive for all higher-leveloperations in the system. In effect, they provide access to the state ofthe data at a particular point in time. As well, since snapshots aretypically implemented using copy-on-write techniques that distinguishwhat has changed from what is resident on disk, these snapshots providedifferences that can also be composed or added together to efficientlycopy data throughout the system. The format of the snapshot may be theformat of data that is copied by Data Mover 502, which is describedbelow.

Standard Copy Operations

When a copy operation is not a snapshot, it may be considered a standardcopy operation. A standard copy operation copies all or a subset of asource data object in one storage pool to a data object in anotherstorage pool. The result is two distinct objects. One type of standardcopy operation that may be used is an initial “baseline” copy. This istypically done when data is initially copied from one Virtual StoragePool into another, such as from a performance-optimized pool to acapacity-optimized storage pool. Another type of standard copy operationmay be used wherein only changed data or differences are copied to atarget storage pool to update the target object. This would occur afteran initial baseline copy has previously been performed.

A complete exhaustive version of an object need not be preserved in thesystem each time a copy is made, even though a baseline copy is neededwhen the Data Virtualization System is first initialized. This isbecause each virtual copy provides access to a complete copy. Any deltaor difference can be expressed in relation to a virtual copy instead ofin relation to a baseline. This has the positive side effect ofvirtually eliminating the common step of walking through a series ofchange lists.

Standard copy operations are initiated by a series of instructions orrequests supplied by the Pool Manager and received by the Data Mover tocause the movement of data among the Data Storage Objects, and tomaintain the Data Storage Objects themselves. The copy operations allowthe creation of copies of the specified Data Storage Objects using theresources of a specified Virtual Storage Pool. The result is a copy ofthe source Data Object in a target Data Object in the storage pool.

The Snapshot and Copy operations are each structured with a preparationoperation and an activation operation. The two steps of prepare andactivate allow the long-running resource allocation operations, typicalof the prepare phase, to be decoupled from the actuation. This isrequired by applications that can only be paused for a short while tofulfill the point-in-time characteristics of a snapshot operation, whichin reality takes a finite but non-zero amount of time to accomplish.Similarly for copy and snapshot operations, this two-step preparationand activation structure allows the Policy Engine to proceed with anoperation only if resources for all of the collection members can beallocated.

Object Maintenance

Object Maintenance operations are a series of operations for maintainingdata objects, including creation, destruction, and duplication. TheObject Manager and Data Mover use functionality provided by a PoolRequest Broker (more below) to implement these operations. The dataobjects may be maintained at a global level, at each Storage Pool, orpreferably both.

Collections

Collection operations are auxiliary functions. Collections are abstractsoftware concepts, lists maintained in memory by the object manager.They allow the Policy Engine 206 to request a series of operations overall of the members in a collection, allowing a consistent application ofa request to all members. The use of collections allows for simultaneousactivation of the point-in-time snapshot so that multiple Data StorageObjects are all captured at precisely the same point in time, as this istypically required by the application for a logically correct restore.The use of collections allows for convenient request of a copy operationacross all members of a collection, where an application would usemultiple storage objects as a logical whole.

Resource Discovery Operations

The Object Manager discovers Virtual Storage Pools by issuing ObjectManagement Operations 505 to the Pool Manager 504, and uses theinformation obtained about each of the pools to select one that meetsthe required criteria for a given request, or in the case where nonematch, a default pool is selected, and the Object Manager can thencreate a data storage object using resources from the selected VirtualStorage Pool.

Mapping and Access

The Object Manager also provides sets of Object Management operations toallow and maintain the availability of these objects to externalApplications. The first set is operations for registering andunregistering the computers where the user's Applications reside. Thecomputers are registered by the identities typical to the storagenetwork in use (e.g. Fibre Channel WWPN, iSCSI identity, etc.). Thesecond set is “mapping” operations, and when permitted by the storagepool from which an object is created, the Data Storage Object can be“mapped,” that is, made available for use to a computer on which a userApplication resides.

This availability takes a form appropriate to the storage, e.g. a blockdevice presented on a SAN as a Fibre Channel disk or iSCSI device on anetwork, a filesystem on a file sharing network, etc. and is usable bythe operating system on the Application computer. Similarly, an“unmapping” operation reverses the availability of the virtual storagedevice on the network to a user Application. In this way, data storedfor one Application, i.e. a backup, can be made available to anotherApplication on another computer at a later time, i.e. a restore.

502 Data Mover

The Data Mover 502 is a software component within the Object Manager andData Mover that reads and writes data among the various Data StorageObjects 503 according to instructions received from the Object Managerfor Snapshot (Point in Time) Copy requests and standard copy requests.The Data Mover provides operations for reading and writing data amonginstances of data objects throughout the system. The Data Mover alsoprovides operations that allow querying and maintaining the state oflong running operations that the Object Manager has requested for it toperform.

The Data Mover uses functionality from the Pool Functionality Providers(see FIG. 6) to accomplish its operation. The Snapshot functionalityprovider 608 allows creation of a data object instance representing aninitial object instance at a specific point in time. The DifferenceEngine functionality provider 614 is used to request a description ofthe differences between two data objects that are related in a temporalchain. For data objects stored on content-addressable pools, a specialfunctionality is provided that can provide differences between any twoarbitrary data objects. This functionality is also provided forperformance-optimized pools, in some cases by an underlying storagevirtualization system, and in other cases by a module that implementsthis on top of commodity storage. The Data Mover 502 uses theinformation about the differences to select the set of data that itcopies between instances of data objects 503.

For a given Pool, the Difference Engine Provider provides a specificrepresentation of the differences between two states of a Data StorageObject over time. For a Snapshot provider the changes between two pointsin time are recorded as writes to a given part of the Data StorageObject. In one embodiment, the difference is represented as a bitmapwhere each bit corresponds to an ordered list of the Data Object areas,starting at the first and ascending in order to the last, where a setbit indicates a modified area. This bitmap is derived from thecopy-on-write bitmaps used by the underlying storage virtualizationsystem. In another embodiment, the difference may be represented as alist of extents corresponding to changed areas of data. For a ContentAddressable storage provider 610, the representation is described below,and is used to determine efficiently the parts of two ContentAddressable Data Objects that differ.

The Data Mover uses this information to copy only those sections thatdiffer, so that a new version of a Data Object can be created from anexisting version by first duplicating it, obtaining the list ofdifferences, and then moving only the data corresponding to thosedifferences in the list. The Data Mover 502 traverses the list ofdifferences, moving the indicated areas from the source Data Object tothe target Data Object. (See Optimal Way for Data Backup and Restore.)

506 Copy Operation—Request Translation and Instructions

The Object Manager 501 instructs the Data Mover 502 through a series ofoperations to copy data among the data objects in the Virtual StoragePools 418. The procedure comprises the following steps, starting at thereception of instructions:

First, create Collection request. A name for the collection is returned.

Second, add Object to Collection. The collection name from above is usedas well as the name of the source Data Object that is to be copied andthe name of two antecedents: a Data Object against which differences areto be taken in the source Storage Resource Pool, and a correspondingData Object in the target Storage Resource Pool. This step is repeatedfor each source Data Object to be operated on in this set.

Third, prepare Copy Request. The collection name is supplied as well asa Storage Resource Pool to act as a target. The prepare commandinstructs the Object Manager to contact the Storage Pool Manager tocreate the necessary target Data Objects, corresponding to each of thesources in the collection. The prepare command also supplies thecorresponding Data Object in the target Storage Resource Pool to beduplicated, so the Provider can duplicate the provided object and usethat as a target object. A reference name for the copy request isreturned.

Fourth, activate Copy Request. The reference name for the copy requestreturned above is supplied. The Data Mover is instructed to copy a givensource object to its corresponding target object. Each request includesa reference name as well as a sequence number to describe the overalljob (the entire set of source target pairs) as well as a sequence numberto describe each individual source-target pair. In addition to thesource-target pair, the names of the corresponding antecedents aresupplied as part of the Copy instruction.

Fifth, the Copy Engine uses the name of the Data Object in the sourcepool to obtain the differences between the antecedent and the sourcefrom the Difference Engine at the source. The indicated differences arethen transmitted from the source to the target. In one embodiment, thesedifferences are transmitted as bitmaps and data. In another embodiment,these differences are transmitted as extent lists and data.

503 Data Storage Objects

Data Storage Objects are software constructs that permit the storage andretrieval of Application data using idioms and methods familiar tocomputer data processing equipment and software. In practice thesecurrently take the form of a SCSI block device on a storage network,e.g. a SCSI LUN, or a content-addressable container, where a designatorfor the content is constructed from and uniquely identifies the datatherein. Data Storage Objects are created and maintained by issuinginstructions to the Pool Manager. The actual storage for persisting theApplication data is drawn from the Virtual Storage Pool from which theData Storage Object is created.

The structure of the data storage object varies depending on the storagepool from which it is created. For the objects that take the form of ablock device on a storage network, the data structure for a given blockdevice Data Object implements a mapping between the Logical BlockAddress (LBA) of each of the blocks within the Data Object to the deviceidentifier and LBA of the actual storage location. The identifier of theData Object is used to identify the set of mappings to be used. Thecurrent embodiment relies on the services provided by the underlyingphysical computer platform to implement this mapping, and relies on itsinternal data structures, such as bitmaps or extent lists.

For objects that take the form of a Content Addressable Container, thecontent signature is used as the identifier, and the Data Object isstored as is described below in the section about deduplication.

504 Pool Manager

A Pool Manager 504 is a software component for managing virtual storageresources and the associated functionality and characteristics asdescribed below. The Object manager 501 and Data Movement Engine 502communicate with one or more Pool Managers 504 to maintain Data StorageObjects 503.

510 Virtual Storage Resources

Virtual Storage Resources 510 are various kinds of storage madeavailable to the Pool Manager for implementing storage pool functions,as described below. In this embodiment, a storage virtualizer is used topresent various external Fibre Channel or iSCSI storage LUNs asvirtualized storage to the Pool Manager 504.

The Storage Pool Manager

FIG. 6 further illustrates the Storage Pool Manager 504. The purpose ofthe storage pool manager is to present underlying virtual storageresources to the Object Manager/Data Mover as Storage Resource Pools,which are abstractions of storage and data management functionality withcommon interfaces that are utilized by other components of the system.These common interfaces typically include a mechanism for identifyingand addressing data objects associated with a specific temporal state,and a mechanism for producing differences between data objects in theform of bitmaps or extents. In this embodiment, the pool managerpresents a Primary Storage Pool, a Performance Optimized Pool, and aCapacity Optimized Pool. The common interfaces allow the object managerto create and delete Data Storage objects in these pools, either ascopies of other data storage objects or as new objects, and the datamover can move data between data storage objects, and can use theresults of data object differencing operations.

The storage pool manager has a typical architecture for implementing acommon interface to diverse implementations of similar functionality,where some functionality is provided by “smart” underlying resources,and other functionality must be implemented on top of less functionalunderlying resources.

Pool request broker 602 and pool functionality providers 604 aresoftware modules executing in either the same process as the ObjectManager/Data Mover, or in another process communicating via a local ornetwork protocol such as TCP. In this embodiment the providers comprisea Primary Storage provider 606, Snapshot provider 608, ContentAddressable provider 610, and Difference Engine provider 614, and theseare further described below. In another embodiment the set of providersmay be a superset of those shown here.

Virtual Storage Resources 510 are the different kinds of storage madeavailable to the Pool Manager for implementing storage pool functions.In this embodiment, the virtual storage resources comprise sets of SCSIlogical units from a storage virtualization system that runs on the samehardware as the pool manager, and accessible (for both data andmanagement operations) through a programmatic interface: in addition tostandard block storage functionality additional capabilities areavailable including creating and deleting snapshots, and trackingchanged portions of volumes. In another embodiment the virtual resourcescan be from an external storage system that exposes similarcapabilities, or may differ in interface (for example accessed through afile-system, or through a network interface such as CIFS, iSCSI orCDMI), in capability (for example, whether the resource supports anoperation to make a copy-on-write snapshot), or in non-functionalaspects (for example, high-speed/limited-capacity such as Solid StateDisk versus low-speed/high-capacity such as SATA disk). The capabilitiesand interface available determine which providers can consume thevirtual storage resources, and which pool functionality needs to beimplemented within the pool manager by one or more providers: forexample, this implementation of a content addressable storage provideronly requires “dumb” storage, and the implementation is entirely withincontent addressable provider 610; an underlying content addressablevirtual storage resource could be used instead with a simpler“pass-through” provider. Conversely, this implementation of a snapshotprovider is mostly “pass-through” and requires storage that exposes aquick point-in-time copy operation.

Pool Request Broker 602 is a simple software component that servicesrequests for storage pool specific functions by executing an appropriateset of pool functionality providers against the configured virtualstorage resource 510. The requests that can be serviced include, but arenot limited to, creating an object in a pool; deleting an object from apool; writing data to an object; reading data from an object; copying anobject within a pool; copying an object between pools; requesting asummary of the differences between two objects in a pool.

Primary storage provider 606 enables management interfaces (for example,creating and deleting snapshots, and tracking changed portions of files)to a virtual storage resource that is also exposed directly toapplications via an interface such as fibre channel, iSCSI, NFS or CIFS.

Snapshot provider 608 implements the function of making a point-in-timecopy of data from a Primary resource pool. This creates the abstractionof another resource pool populated with snapshots. As implemented, thepoint-in-time copy is a copy-on-write snapshot of the object from theprimary resource pool, consuming a second virtual storage resource toaccommodate the copy-on-write copies, since this managementfunctionality is exposed by the virtual storage resources used forprimary storage and for the snapshot provider.

Difference engine provider 614 can satisfy a request for two objects ina pool to be compared that are connected in a temporal chain. Thedifference sections between the two objects are identified andsummarized in a provider-specific way, e.g. using bitmaps or extents.For example, the difference sections might be represented as a bitmapwhere each set bit denotes a fixed size region where the two objectsdiffer; or the differences might be represented procedurally as a seriesof function calls or callbacks.

Depending on the virtual storage resource on which the pool is based, oron other providers implementing the pool, a difference engine mayproduce a result efficiently in various ways. As implemented, adifference engine acting on a pool implemented via a snapshot provideruses the copy-on-write nature of the snapshot provider to track changesto objects that have had snapshots made. Consecutive snapshots of asingle changing primary object thus have a record of the differencesthat is stored alongside them by the snapshot provider, and thedifference engine for snapshot pools simply retrieves this record ofchange. Also as implemented, a difference engine acting on a poolimplemented via a Content Addressable provider uses the efficient treestructure (see below, FIG. 12) of the content addressable implementationto do rapid comparisons between objects on demand.

Content addressable provider 610 implements a write-once contentaddressable interface to the virtual storage resource it consumes. Itsatisfies read, write, duplicate and delete operations. Each written orcopied object is identified by a unique handle that is derived from itscontent. The content addressable provider is described further below(FIG. 11).

Pool Manager Operations

In operation, the pool request broker 502 accepts requests for datamanipulation operations such as copy, snapshot, or delete on a pool orobject. The request broker determines which provider code from pool 504to execute by looking at the name or reference to the pool or object.The broker then translates the incoming service request into a form thatcan be handled by the specific pool functionality provider, and invokesthe appropriate sequence of provider operations.

For example, an incoming request could ask to make a snapshot from avolume in a primary storage pool, into a snapshot pool. The incomingrequest identifies the object (volume) in the primary storage pool byname, and the combination of name and operation (snapshot) determinesthat the snapshot provider should be invoked which can makepoint-in-time snapshots from the primary pool using the underlyingsnapshot capability. This snapshot provider will translate the requestinto the exact form required by the native copy-on-write functionperformed by the underlying storage virtualization appliance, such asbitmaps or extents, and it will translate the result of the nativecopy-on-write function to a storage volume handle that can be returnedto the object manager and used in future requests to the pool manager.

Optimal Way for Data Backup Using the Object Manager and Data Mover

Optimal Way for Data Backup is a series of operations to make successiveversions of Application Data objects over time, while minimizing theamount of data that must be copied by using bitmaps, extents and othertemporal difference information stored at the Object Mover. It storesthe application data in a data storage object and associates with it themetadata that relates the various changes to the application data overtime, such that changes over time can be readily identified.

In a preferred embodiment, the procedure comprises the following steps:

-   -   1. The mechanism provides an initial reference state, e.g. T0,        of the Application Data within a Data Storage Object.    -   2. Subsequent instances (versions) are created on demand over        time of the Data Storage Object in a Virtual Storage Pool that        has a Difference Engine Provider.    -   3. Each successive version, e.g. T4, T5, uses the Difference        Engine Provider for the Virtual Storage Pool to obtain the        difference between it and the instance created prior to it, so        that T5 is stored as a reference to T4 and a set of differences        between T5 and T4.    -   4. The Copy Engine receives a request to copy data from one data        object (the source) to another data object (the destination).    -   5. If the Virtual Storage Pool in which the destination object        will be created contains no other objects created from prior        versions of the source data object, then a new object is created        in the destination Virtual Storage Pool and the entire contents        of the source data object are copied to the destination object;        the procedure is complete. Otherwise the next steps are        followed.    -   6. If the Virtual Storage Pool in which the destination object        is created contains objects created from prior versions of the        source data object, a recently created prior version in the        destination Virtual Storage Pool is selected for which there        exists a corresponding prior version in the Virtual Storage Pool        of the source data object. For example, if a copy of T5 is        initiated from a snapshot pool, and an object created at time T3        is the most recent version available at the target, T3 is        selected as the prior version.    -   7. Construct a time-ordered list of the versions of the source        data object, beginning with an initial version identified in the        previous step, and ending with the source data object that is        about to be copied. In the above example, at the snapshot pool,        all states of the object are available, but only the states        including and following T3 are of interest: T3, T4, T5.    -   8. Construct a corresponding list of the differences between        each successive version in the list such that all of the        differences, from the beginning version of the list to the end        are represented. Difference both, identify which portion of data        has changed and includes the new data for the corresponding        time. This creates a set of differences from the target version        to the source version, e.g. the difference between T3 and T5.    -   9. Create the destination object by duplicating the prior        version of the object identified in Step 6 in the destination        Virtual Storage Pool, e.g. object T3 in the target store.    -   10. Copy the set of differences identified in the list created        in Step 8 from the source data object to the destination object;        the procedure is complete.

Each data object within the destination Virtual Storage Pool iscomplete; that is, it represents the entire data object and allowsaccess to the all of the Application Data at the point in time withoutrequiring external reference to state or representations at other pointsin time. The object is accessible without replaying all deltas from abaseline state to the present state. Furthermore, the duplication ofinitial and subsequent versions of the data object in the destinationVirtual Storage Pool does not require exhaustive duplication of theApplication Data contents therein. Finally, to arrive at second andsubsequent states requires only the transmission of the changes trackedand maintained, as described above, without exhaustive traversal,transmission or replication of the contents of the data storage object.

Optimal Way for Data Restore Using the Object Manager and Data Mover

Intuitively, the operation of the Optimal Way for Data Restore is theconverse of the Optimal Way for Data Backup. The procedure to recreatethe desired state of a data object in a destination Virtual Storage Poolat a given point in time comprises the following steps:

-   1. Identify a version of the data object in another Virtual Storage    Pool that has a Difference Engine Provider, corresponding to the    desired state to be recreated. This is the source data object in the    source Virtual Storage Pool.-   2. Identify a preceding version of the data object to be recreated    in the destination Virtual Storage Pool.-   3. If no version of the data object is identified in Step 2, then    create a new destination object in the destination Virtual Storage    Pool and copy the data from the source data object to the    destination data object. The procedure is complete. Otherwise,    proceed with the following steps.-   4. If a version of the data object is identified in Step 2, then    identify a data object in the source Virtual Storage Pool    corresponding to the data object identified in Step 2.-   5. If no data object is identified in Step 4, then create a new    destination object in the destination Virtual Storage Pool and copy    the data from the source data object to the destination data object.    The procedure is complete. Otherwise, proceed with the following    steps.-   6. Create a new destination data object in the Destination Virtual    Storage Pool by duplicating the data object identified in Step 2.-   7. Employ the Difference Engine Provider for the source Virtual    Storage Pool to obtain the set of differences between the data    object identified in Step 1 and the data object identified in Step    4.-   8. Copy the data identified by the list created in Step 7 from the    source data object to the destination data object. The procedure is    complete.

Access to the desired state is complete: it does not require externalreference to other containers or other states. Establishing the desiredstate given a reference state requires neither exhaustive traversal norexhaustive transmission, only the retrieved changes indicated by theprovided representations within the source Virtual Storage Pool.

The Service Level Agreement

FIG. 7 illustrates the Service Level Agreement. The Service LevelAgreement captures the detailed business requirements with respect tosecondary copies of the application data. In the simplest description,the business requirements define when and how often copies are created,how long they are retained and in what type of storage pools thesecopies reside. This simplistic description does not capture severalaspects of the business requirements. The frequency of copy creation fora given type of pool may not be uniform across all hours of the day oracross all days of a week. Certain hours of the day, or certain days ofa week or month may represent more (or less) critical periods in theapplication data, and thus may call for more (or less) frequent copies.Similarly, all copies of application data in a particular pool may notbe required to be retained for the same length of time. For example, acopy of the application data created at the end of monthly processingmay need to be retained for a longer period of time than a copy in thesame storage pool created in the middle of a month.

The Service Level Agreement 304 of certain embodiments has been designedto represent all of these complexities that exist in the businessrequirements. The Service Level Agreement has four primary parts: thename, the description, the housekeeping attributes and a collection ofService Level Policies. As mentioned above, there is one SLA perapplication.

The name attribute 701 allows each Service Level Agreement to have aunique name.

The description attribute 702 is where the user can assign a helpfuldescription for the Service Level Agreement.

The Service Level agreement also has a number of housekeeping attributes703 that enable it to be maintained and revised. These attributesinclude but are not limited to the owner's identity, the dates and timesof creation, modification and access, priority, enable/disable flags.

The Service Level Agreement also contains a plurality of Service LevelPolicies 705. Some Service level Agreements may have just a singleService Level Policy. More typically, a single SLA may contain tens ofpolicies.

Each Service Level Policy consists of at least the following, in certainembodiments: the source storage pool location 706 and type 708; thetarget storage pool location 710 and type 712; the frequency for thecreation of copies 714, expressed as a period of time; the length ofretention of the copy 716, expressed as a period of time; the hours ofoperation 718 during the day for this particular Service Level Policy;and the days of the week, month or year 720 on which this Service LevelPolicy applies.

Each Service Level Policy specifies a source and target storage pool,and the frequency of copies of application data that are desired betweenthose storage pools. Furthermore, the Service Level Policy specifies itshours of operation and days on which it is applicable. Each ServiceLevel Policy is the representation of one single statement in thebusiness requirements for the protection of application data. Forexample, if a particular application has a business requirement for anarchive copy to be created each month after the monthly close andretained for three years, this might translate to a Service level Policythat requires a copy from the Local Backup Storage Pool into theLong-term Archive Storage Pool at midnight on the last day of the month,with a retention of three years.

All of the Service Level Policies with a particular combination ofsource and destination pool and location, say for example, sourcePrimary Storage pool and destination local Snapshot pool, when takentogether, specify the business requirements for creating copies intothat particular destination pool. Business requirements may dictate forexample that snapshot copies be created every hour during regularworking hours, but only once every four hours outside of these times.Two Service Level Policies with the same source and target storage poolswill effectively capture these requirements in a form that can be putinto practice by the Service Policy Engine.

This form of a Service Level Agreement allows the representation of theschedule of daily, weekly and monthly business activities, and thuscaptures business requirements for protecting and managing applicationdata much more accurately than traditional RPO and RPO based schemes. Byallowing hour of operation and days, weeks, and months of the year,scheduling can occur on a “calendar basis.”

Taken together, all of the Service Level Policies with one particularcombination of source and destinations, for example, “source: localprimary and destination: local performance optimized”, captures thenon-uniform data protection requirements for one type of storage. Asingle RPO number, on the other hand, forces a single uniform frequencyof data protection across all times of day and all days. For example, acombination of Service Level Policies may require a large number ofsnapshots to be preserved for a short time, such as 10 minutes, and alesser number of snapshots to be preserved for a longer time, such as 8hours; this allows a small amount of information that has beenaccidentally deleted can be reverted to a state not more than 10 minutesbefore, while still providing substantial data protection at longer timehorizons without requiring the storage overhead of storing all snapshotstaken every ten minutes. As another example, the backup data protectionfunction may be given one Policy that operates with one frequency duringthe work week, and another frequency during the weekend.

When Service Level Policies for all of the different classes of sourceand destination storage are included, the Service Level Agreement fullycaptures all of the data protection requirements for the entireapplication, including local snapshots, local long duration stores,off-site storage, archives, etc. A collection of policies within a SLAis capable of expressing when a given function should be performed, andis capable of expressing multiple data management functions that shouldbe performed on a given source of data.

Service Level Agreements are created and modified by the user through auser interface on a management workstation. These agreements areelectronic documents stored by the Service Policy Engine in a structuredSQL database or other repository that it manages. The policies areretrieved, electronically analyzed, and acted upon by the Service PolicyEngine through its normal scheduling algorithm as described below.

FIG. 8 illustrates the Application Specific Module 402. The ApplicationSpecific module runs close to the Application 300 (as described above),and interacts with the Application and its operating environment togather metadata and to query and control the Application as required fordata management operations.

The Application Specific Module interacts with various components of theapplication and its operating environment including Application ServiceProcesses and Daemons 801, Application Configuration Data 802, OperatingSystem Storage Services 803 (such as VSS and VDS on Windows), LogicalVolume Management and Filesystem Services 804, and Operating SystemDrivers and Modules 805.

The Application Specific Module performs these operations in response tocontrol commands from the Service Policy Engine 406. There are twopurposes for these interactions with the application: MetadataCollection and Application Consistency.

Metadata Collection is the process by which the Application SpecificModule collects metadata about the application. In some embodiments,metadata includes information such as: configuration parameters for theapplication; state and status of the application; control files andstartup/shutdown scripts for the application; location of the datafiles,journal and transaction logs for the application; and symbolic links,filesystem mount points, logical volume names, and other such entitiesthat can affect the access to application data.

Metadata is collected and saved along with application data and SLAinformation. This guarantees that each copy of application data withinthe system is self contained and includes all of the details required torebuild the application data.

Application Consistency is the set of actions that ensure that when acopy of the application data is created, the copy is valid, and can berestored into a valid instance of the application. This is critical whenthe business requirements dictate that the application be protectedwhile it is live, in its online, operational state. The application mayhave interdependent data relations within its data stores, and if theseare not copied in a consistent state will not provide a valid restorableimage.

The exact process of achieving application consistency varies fromapplication to application. Some applications have a simple flushcommand that forces cached data to disk. Some applications support a hotbackup mode where the application ensures that its operations arejournalled in a manner that guarantees consistency even as applicationdata is changing. Some applications require interactions with operatingsystem storage services such as VSS and VDS to ensure consistency. TheApplication Specific Module is purpose-built to work with a particularapplication and to ensure the consistency of that application. TheApplication Specific Module interacts with the underlying storagevirtualization device and the Object Manager to provide consistentsnapshots of application data.

For efficiency, the preferred embodiment of the Application SpecificModule 402 is to run on the same server as Application 300. This assuresthe minimum latency in the interactions with the application, andprovides access to storage services and filesystems on the applicationhost. The application host is typically considered primary storage,which is then snapshotted to a performance-optimized store.

In order to minimize interruption of a running application, includingminimizing preparatory steps, the Application Specific Module is onlytriggered to make a snapshot when access to application data is requiredat a specific time, and when a snapshot for that time does not existelsewhere in the system, as tracked by the Object Manager. By trackingwhich times snapshots have been made, the Object Manager is able tofulfill subsequent data requests from the performance-optimized datastore, including for satisfying multiple requests for backup andreplication which may issue from secondary, capacity-optimized pools.The Object Manager may be able to provide object handles to the snapshotin the performance-optimized store, and may direct theperformance-optimized store in a native format that is specific to theformat of the snapshot, which is dependent on the underlying storageappliance. In some embodiments this format may be application datacombined with one or more LUN bitmaps indicating which blocks havechanged; in other embodiments it may be specific extents. The formatused for data transfer is thus able to transfer only a delta ordifference between two snapshots using bitmaps or extents.

Metadata, such as the version number of the application, may also bestored for each application along with the snapshot. When a SLA policyis executed, application metadata is read and used for the policy. Thismetadata is stored along with the data objects. For each SLA,application metadata will only be read once during the lightweightsnapshot operation, and preparatory operations which occur at that timesuch as flushing caches will only be performed once during thelightweight snapshot operation, even though this copy of applicationdata along with its metadata may be used for multiple data managementfunctions.

The Service Policy Engine

FIG. 9 illustrates the Service Policy Engine 406. The Service PolicyEngine contains the Service Policy Scheduler 902, which examines all ofthe Service Level Agreements configured by the user and makes schedulingdecisions to satisfy Service Level Agreements. It relies on several datastores to capture information and persist it over time, including, insome embodiments, a SLA Store 904, where configured Service LevelAgreements are persisted and updated; a Resource Profile Store 906,storing Resource Profiles that provide a mapping between logical storagepool names and actual storage pools; Protection Catalog Store 908, whereinformation is cataloged about previous successful copies created invarious pools that have not yet expired; and centralized History Store910.

History Store 910 is where historical information about past activitiesis saved for the use of all data management applications, including thetimestamp, order and hierarchy of previous copies of each applicationinto various storage pools. For example, a snapshot copy from a primarydata store to a capacity-optimized data store that is initiated at 1P.M. and is scheduled to expire at 9 P.M. will be recorded in HistoryStore 910 in a temporal data store that also includes linked object datafor snapshots for the same source and target that have taken place at 11A.M. and 12 P.M.

These stores are managed by the Service Policy Engine. For example, whenthe user, through the Management workstation creates a Service LevelAgreement, or modifies one of the policies within it, it is the ServicePolicy Engine that persists this new SLA in its store, and reacts tothis modification by scheduling copies as dictated by the SLA.Similarly, when the Service Policy Engine successfully completes a datamovement job that results in a new copy of an application in a StoragePool, the Storage Policy Engine updates the History Store, so that thiscopy will be factored into future decisions.

The preferred embodiment of the various stores used by the ServicePolicy Engine is in the form of tables in a relational databasemanagement system in close proximity to the Service Policy Engine. Thisensures consistent transactional semantics when querying and updatingthe stores, and allows for flexibility in retrieving interdependentdata.

The scheduling algorithm for the Service Policy Scheduler 902 isillustrated in FIG. 10. When the Service Policy Scheduler decides itneeds to make a copy of application data from one storage pool toanother, it initiates a Data Movement Requestor and Monitor task, 912.These tasks are not recurring tasks and terminate when they arecompleted. Depending on the way that Service Level Policies arespecified, a plurality of these requestors might be operational at thesame time.

The Service Policy Scheduler considers the priorities of Service LevelAgreements when determining which additional tasks to undertake. Forexample, if one Service Level Agreement has a high priority because itspecifies the protection for a mission-critical application, whereasanother SLA has a lower priority because it specifies the protection fora test database, then the Service Policy Engine may choose to run onlythe protection for the mission-critical application, and may postpone oreven entirely skip the protection for the lower priority application.This is accomplished by the Service Policy Engine scheduling a higherpriority SLA ahead of a lower priority SLA. In the preferred embodiment,in such a situation, for auditing purposes, the Service Policy Enginewill also trigger a notification event to the management workstation.

The Policy Scheduling Algorithm

FIG. 10 illustrates the flowchart of the Policy Schedule Engine. ThePolicy Schedule Engine continuously cycles through all the SLAs defined.When it gets to the end of all of the SLAs, it sleeps for a short while,e.g. 10 seconds, and resumes looking through the SLAs again. Each SLAencapsulates the complete data protection business requirements for oneapplication; thus all of the SLAs represent all of the applications.

For each SLA, the schedule engine collects together all of the ServiceLevel Policies that have the same source pool and destination pool 1004the process state at 1000 and iterates to the next SLA in the set ofSLAs in 1002. Taken together, this subset of the Service Level Policiesrepresent all of the requirements for a copy from that source storagepool to that particular destination storage pool.

Among this subset of Service Level Policies, the Service PolicyScheduler discards the policies that are not applicable to today, or areoutside their hours of operation. Among the policies that are left, findthe policy that has the shortest frequency 1006, and based on thehistory data and in history store 910, the one with the longestretention that needs to be run next 1008.

Next, there are a series of checks 1010-1014 which rule out making a newcopy of application data at this time—because the new copy is not yetdue, because a copy is already in progress or because there is not newdata to copy. If any of these conditions apply, the Service PolicyScheduler moves to the next combination of source and destination pools1004. If none of these conditions apply, a new copy is initiated. Thecopy is executed as specified in the corresponding service level policywithin this SLA 1016.

Next, the Scheduler moves to the next Source and Destination poolcombination for the same Service Level agreement 1018. If there are nomore distinct combinations, the Scheduler moves on to the next ServiceLevel Agreement 1020.

After the Service Policy Scheduler has been through allsource/destination pool combinations of all Service Level Agreements, itpauses for a short period and then resumes the cycle.

A simple example system with a snapshot store and a backup store, withonly 2 policies defined, would interact with the Service PolicyScheduler as follows. Given two policies, one stating “backup everyhour, the backup to be kept for 4 hours” and another stating “backupevery 2 hours, the backup to be kept for 8 hours,” the result would be asingle snapshot taken each hour, the snapshots each being copied to thebackup store but retained a different amount of time at both thesnapshot store and the backup store. The “backup every 2 hours” policyis scheduled to go into effect at 12:00 P.M by the system administrator.

At 4:00 P.M., when the Service Policy Scheduler begins operating at step1000, it finds the two policies at step 1002. (Both policies applybecause a multiple of two hours has elapsed since 12:00 P.M.) There isonly one source and destination pool combination at step 1004. There aretwo frequencies at step 1006, and the system selects the 1-hourfrequency because it is shorter than the 2-hour frequency. There are twooperations with different retentions at step 1008, and the systemselects the operation with the 8-hour retention, as it has the longerretention value. Instead of one copy being made to satisfy the 4-hourrequirement and another copy being made to satisfy the 8-hourrequirement, the two requirements are coalesced into the longer 8-hourrequirement, and are satisfied by a single snapshot copy operation. Thesystem determines that a copy is due at step 1010, and checks therelevant objects at the History Store 910 to determine if the copy hasalready been made at the target (at step 912) and at the source (at step914). If these checks are passed, the system initiates the copy at step916, and in the process triggers a snapshot to be made and saved at thesnapshot store. The snapshot is then copied from the snapshot store tothe backup store. The system then goes to sleep 1022 and wakes up againafter a short period, such as 10 seconds. The result is a copy at thebackup store and a copy at the snapshot store, where every even-hoursnapshot lasts for 8 hours, and every odd-hour snapshot lasts 4 hours.The even-hour snapshots at the backup store and the snapshot store areboth tagged with the retention period of 8 hours, and will beautomatically deleted from the system by another process at that time.

Note that there is no reason to take two snapshots or make two backupcopies at 2 o'clock, even though both policies apply, because bothpolicies are satisfied by a single copy. Combining and coalescing thesesnapshots results in the reduction of unneeded operations, whileretaining the flexibility of multiple separate policies. As well, it maybe helpful to have two policies active at the same time for the sametarget with different retention. In the example given, there are morehourly copies kept than two-hour copies, resulting in more granularityfor restore at times that are closer to the present. For example, in theprevious system, if at 7:30 P.M. damage is discovered from earlier inthe afternoon, a backup will be available for every hour for the pastfour hours: 4, 5, 6, 7 P.M. As well, two more backups will have beenretained from 2 P.M. and 12 P.M.

The Content Addressable Store

FIG. 11 is a block diagram of the modules implementing the contentaddressable store for the Content Addressable Provider 510.

The content addressable store 510 implementation provides a storageresource pool that is optimized for capacity rather than for copy-in orcopy-out speed, as would be the case for the performance-optimized poolimplemented through snapshots, described earlier, and thus is typicallyused for offline backup, replication and remote backup. Contentaddressable storage provides a way of storing common subsets ofdifferent objects only once, where those common subsets may be ofvarying sizes but typically as small as 4 KiBytes. The storage overheadof a content addressable store is low compared to a snapshot store,though the access time is usually higher. Generally objects in a contentaddressable store have no intrinsic relationship to one another, eventhough they may share a large percentage of their content, though inthis implementation a history relationship is also maintained, which isan enabler of various optimizations to be described. This contrasts witha snapshot store where snapshots intrinsically form a chain, eachstoring just deltas from a previous snapshot or baseline copy. Inparticular, the content addressable store will store only one copy of adata subset that is repeated multiple times within a single object,whereas a snapshot-based store will store at least one full-copy of anyobject.

The content addressable store 510 is a software module that executes onthe same system as the pool manager, either in the same process or in aseparate process communicating via a local transport such as TCP. Inthis embodiment, the content addressable store module runs in a separateprocess so as to minimize impact of software failures from differentcomponents.

This module's purpose is to allow storage of Data Storage Objects 403 ina highly space-efficient manner by deduplicating content (i.e., ensuringrepeated content within single or multiple data objects is stored onlyonce).

The content addressable store module provides services to the poolmanager via a programmatic API. These services comprise the following:

Object to Handle mapping 1102: an object can be created by writing datainto the store via an API; once the data is written completely the APIreturns an object handle determined by the content of the object.Conversely, data may be read as a stream of bytes from an offset withinan object by providing the handle. Details of how the handle isconstructed are explained in connection with the description of FIG. 12.

Temporal Tree Management 1104 tracks parent/child relationships betweendata objects stored. When a data object is written into the store 510,an API allows it to be linked as a child to a parent object already inthe store. This indicates to the content addressable store that thechild object is a modification of the parent. A single parent may havemultiple children with different modifications, as might be the case forexample if an application's data were saved into the store regularly forsome while; then an early copy were restored and used as a new startingpoint for subsequent modifications. Temporal tree management operationsand data models are described in more detail below.

Difference Engine 1106 can generate a summary of difference regionsbetween two arbitrary objects in the store. The differencing operationis invoked via an API specifying the handles of two objects to becompared, and the form of the difference summary is a sequence ofcallbacks with the offset and size of sequential difference sections.The difference is calculated by comparing two hashed representations ofthe objects in parallel.

Garbage Collector 1108 is a service that analyzes the store to findsaved data that is not referenced by any object handle, and to reclaimthe storage space committed to this data. It is the nature of thecontent addressable store that much data is referenced by multipleobject handles, i.e., the data is shared between data objects; some datawill be referenced by a single object handle; but data that isreferenced by no object handles (as might be the case if an objecthandle has been deleted from the content addressable system) can besafely overwritten by new data.

Object Replicator 1110 is a service to duplicate data objects betweentwo different content addressable stores. Multiple content addressablestores may be used to satisfy additional business requirements, such asoffline backup or remote backup.

These services are implemented using the functional modules shown inFIG. 11. The Data Hash module 1112 generates fixed length keys for datachunks up to a fixed size limit. For example, in this embodiment themaximum size of chunk that the hash generator will make a key for is 64KiB. The fixed length key is either a hash, tagged to indicate thehashing scheme used, or a non-lossy algorithmic encoding. The hashingscheme used in this embodiment is SHA-1, which generates a securecryptographic hash with a uniform distribution and a probability of hashcollision near enough zero that no facility need be incorporated intothis system to detect and deal with collisions.

The Data Handle Cache 1114 is a software module managing an in-memorydatabase that provides ephemeral storage for data and for handle-to-datamappings.

The Persistent Handle Management Index 1104 is a reliable persistentdatabase of CAH-to-data mappings. In this embodiment it is implementedas a B-tree, mapping hashes from the hash generator to pages in thepersistent data store 1118 that contain the data for this hash. Sincethe full B-tree cannot be held in memory at one time, for efficiency,this embodiment also uses an in-memory bloom filter to avoid expensiveB-tree searches for hashes known not to be present.

The Persistent Data Storage module 1118 stores data and handles tolong-term persistent storage, returning a token indicating where thedata is stored. The handle/token pair is subsequently used to retrievethe data. As data is written to persistent storage, it passes through alayer of lossless data compression 1120, in this embodiment implementedusing zlib, and a layer of optional reversible encryption 1122, which isnot enabled in this embodiment.

For example, copying a data object into the content addressable store isan operation provided by the object/handle mapper service, since anincoming object will be stored and a handle will be returned to therequestor. The object/handle mapper reads the incoming object, requestshashes to be generated by the Data Hash Generator, stores the data toPersistent Data Storage and the handle to the Persistent HandleManagement Index. The Data Handle Cache is kept updated for future quicklookups of data for the handle. Data stored to Persistent Data Storageis compressed and (optionally) encrypted before being written to disk.Typically a request to copy in a data object will also invoke thetemporal tree management service to make a history record for theobject, and this is also persisted via Persistent Data Storage.

As another example, copying a data object out of the content addressablestore given its handle is another operation provided by theobject/handle mapper service. The handle is looked up in the Data HandleCache to locate the corresponding data; if the data is missing in thecache the persistent index is used; once the data is located on disk, itis retrieved via persistent data storage module (which decrypts anddecompresses the disk data) and then reconstituted to return to therequestor.

The Content Addressable Store Handle

FIG. 12 shows how the handle for a content addressed object isgenerated. The data object manager references all content addressableobjects with a content addressable handle. This handle is made up ofthree parts. The first part 1201 is the size of the underlying dataobject the handle immediately points to. The second part 1202 is thedepth of object it points to. The third 1203 is a hash of the object itpoints to. Field 1203 optionally includes a tag indicating that the hashis a non-lossy encoding of the underlying data. The tag indicates theencoding scheme used, such as a form of run-length encoding (RLE) ofdata used as an algorithmic encoding if the data chunk can be fullyrepresented as a short enough RLE. If the underlying data object is toolarge to be represented as a non-lossy encoding, a mapping from the hashto a pointer or reference to the data is stored separately in thepersistent handle management index 1104.

The data for a content addressable object is broken up into chunks 1204.The size of each chunk must be addressable by one content addressablehandle 1205. The data is hashed by the data hash module 1102, and thehash of the chunk is used to make the handle. If the data of the objectfits in one chunk, then the handle created is the final handle of theobject. If not, then the handles themselves are grouped together intochunks 1206 and a hash is generated for each group of handles. Thisgrouping of handles continues 1207 until there is only one handle 1208produced which is then the handle for the object.

When an object is to be reconstituted from a content handle (thecopy-out operation for the storage resource pool), the top level contenthandle is dereferenced to obtain a list of next-level content handles.These are dereferenced in turn to obtain further lists of contenthandles until depth-0 handles are obtained. These are expanded to data,either by looking up the handle in the handle management index or cache,or (in the case of an algorithmic hash such as run-length encoding)expanding deterministically to the full content.

Temporal Tree Management

FIG. 13 illustrates the temporal tree relationship created for dataobjects stored within the content addressable store. This particulardata structure is utilized only within the content addressable store.The temporal tree management module maintains data structures 1302 inthe persistent store that associate each content-addressed data objectto a parent (which may be null, to indicate the first in a sequence ofrevisions). The individual nodes of the tree contain a single hashvalue. This hash value references a chunk of data, if the hash is adepth-0 hash, or a list of other hashes, if the hash is a depth-1 orhigher hash. The references mapped to a hash value is contained in thePersistent Handle Management Index 1104. In some embodiments the edgesof the tree may have weights or lengths, which may be used in analgorithm for finding neighbors.

This is a standard tree data structure and the module supports standardmanipulation operations, in particular: 1310 Add: adding a leaf below aparent, which results in a change to the tree as between initial state1302 and after-add state 1304; and 1312 Remove: removing a node (andreparenting its children to its parent), which results in a change tothe tree as between after-add state 1304 and after-remove state 1306.

The “Add” operation is used whenever an object is copied-in to the CASfrom an external pool. If the copy-in is via the Optimal Way for DataBackup, or if the object is originating in a different CAS pool, then itis required that a predecessor object be specified, and the Addoperation is invoked to record this predecessor/successor relationship.

The “Remove” operation is invoked by the object manager when the policymanager determines that an object's retention period has expired. Thismay lead to data stored in the CAS having no object in the temporal treereferring to it, and therefore a subsequent garbage collection pass canfree up the storage space for that data as available for re-use.

Note that it is possible for a single predecessor to have multiplesuccessors or child nodes. For example, this may occur if an object isoriginally created at time T1 and modified at time T2, the modificationsare rolled back via a restore operation, and subsequent modificationsare made at time T3. In this example, state T1 has two children, stateT2 and state T3.

Different CAS pools may be used to accomplish different businessobjectives such as providing disaster recovery in a remote location.When copying from one CAS to another CAS, the copy may be sent as hashesand offsets, to take advantage of the native deduplication capabilitiesof the target CAS. The underlying data pointed to by any new hashes isalso sent on an as-needed basis.

The temporal tree structure is read or navigated as part of theimplementation of various services:

-   -   Garbage Collection navigates the tree in order to reduce the        cost of the “mark” phase, as described below    -   Replication to a different CAS pool finds a set of        near-neighbors in the temporal tree that are also known to have        been transferred already to the other CAS pool, so that only a        small set of differences need to be transferred additionally    -   Optimal-Way for data restore uses the temporal tree to find a        predecessor that can be used as a basis for the restore        operation. In the CAS temporal tree data structure, children are        subsequent versions, e.g., as dictated by archive policy.        Multiple children are supported on the same parent node; this        case may arise when a parent node is changed, then used as the        basis for a restore, and subsequently changed again.        CAS Difference Engine

The CAS difference engine 1106 compares two objects identified by hashvalues or handles as in FIGS. 11 and 12, and produces a sequence ofoffsets and extents within the objects where the object data is known todiffer. This sequence is achieved by traversing the two object trees inparallel in the hash data structure of FIG. 12. The tree traversal is astandard depth- or breadth-first traversal. During traversal, the hashesat the current depth are compared. Where the hash of a node is identicalbetween both sides, there is no need to descend the tree further, so thetraversal may be pruned. If the hash of a node is not identical, thetraversal continues descending into the next lowest level of the tree.If the traversal reaches a depth-0 hash that is not identical to itscounterpart, then the absolute offset into the data object beingcompared where the non-identical data occurs, together with the datalength, is emitted into the output sequence. If one object is smaller insize than another, then its traversal will complete earlier, and allsubsequent offsets encountered in the traversal of the other are emittedas differences.

Garbage Collection Via Differencing

As described under FIG. 11, Garbage Collector is a service that analyzesa particular CAS store to find saved data that is not referenced by anyobject handle in the CAS store temporal data structure, and to reclaimthe storage space committed to this data. Garbage collection uses astandard “Mark and Sweep” approach. Since the “mark” phase may be quiteexpensive, the algorithm used for the mark phase attempts to minimizemarking the same data multiple times, even though it may be referencedmany times; however the mark phase must be complete, ensuring that noreferenced data is left unmarked, as this would result in data loss fromthe store as, after a sweep phase, unmarked data would later beoverwritten by new data.

The algorithm employed for marking referenced data uses the fact thatobjects in the CAS are arranged in graphs with temporal relationshipsusing the data structure depicted in FIG. 13. It is likely that objectsthat share an edge in these graphs differ in only a small subset oftheir data, and it is also rare that any new data chunk that appearswhen an object is created from a predecessor should appear again betweenany two other objects. Thus, the mark phase of garbage collectionprocesses each connected component of the temporal graph.

FIG. 14 is an example of garbage collection using temporal relationshipsin certain embodiments. A depth-first search is made, represented byarrows 1402, of a data structure containing temporal relationships. Takea starting node 1404 from which to begin the tree traversal. Node 1404is the tree root and references no objects. Node 1406 containsreferences to objects H₁ and H₂, denoting a hash value for object 1 anda hash value for object 2. All depth-0, depth-1 and higher data objectsthat are referenced by node 1406, here H₁ and H₂, are enumerated andmarked as referenced.

Next, node 1408 is processed. As it shares an edge with node 1406, whichhas been marked, the difference engine is applied to the differencebetween the object referenced by 1406 and the object referenced by 1408,obtaining a set of depth-0, depth-1 and higher hashes that exist in theunmarked object but not in the marked object. In the figure, the hashthat exists in node 1408 but not in node 1406 is H₃, so H₃ is marked asreferenced. This procedure is continued until all edges are exhausted.

A comparison of the results produced by a prior art algorithm 1418 andthe present embodiment 1420 shows that when node 1408 is processed bythe prior art algorithm, previously-seen hashes H₁ and H₂ are emittedinto the output stream along with new hash H₃. Present embodiment 1420does not emit previously seen hashes into the output stream, resultingin only new hashes H₃, H₄, H₅, H₆, H₇ being emitted into the outputstream, with a corresponding improvement in performance. Note that thismethod does not guarantee that data will not be marked more than once.For example, if hash value H₄ occurs independently in node 1416, it willbe independently marked a second time.

Copy an Object into the CAS

Copying an object from another pool into the CAS uses the softwaremodules described in FIG. 11 to produce a data structure referenced byan object handle as in FIG. 12. The input to the process is (a) asequence of chunks of data at specified offsets, sized appropriately formaking depth-0 handles, and optionally (b) a previous version of thesame object. Implicitly, the new object will be identical to theprevious version except where the input data is provided and itselfdiffers from the previous version. The algorithm for the copy-inoperation is illustrated in a flowchart at FIG. 15.

If a previous version (b) is provided, then the sequence (a) may be asparse set of changes from (b). In the case that the object to be copiedand is known to differ from a previous object at only a few points, thiscan greatly reduce the amount of data that needs to be copied in, andtherefore reduce the computation and i/o activity required. This is thecase, for example, when the object is to be copied in via the optimalway for data backup described previously.

Even if the sequence (a) includes sections that are largely unchangedfrom a predecessor, identifying the predecessor (b) allows the copy-inprocedure to do quick checks as to whether the data has indeed changedand therefore to avoid data duplication at a finer level of granularitythan might be possible for the difference engine in some other storagepool providing input to a CAS.

Implicitly then, the new object will be identical to the previousversion except where the input data is provided and itself differs fromthe previous version. The algorithm for the copy-in operation isillustrated in a flowchart at FIG. 15.

The process starts at step 1500 as an arbitrarily-sized data object inthe temporal store is provided, and proceeds to 1502, which enumeratesany and all hashes (depth-0 through the highest level) referenced by thehash value in the predecessor object, if such is provided. This will beused as a quick check to avoid storing data that is already contained inthe predecessor.

At step 1504, if a predecessor is input, create a reference to a cloneof it in the content-addressable data store temporal data structure.This clone will be updated to become the new object. Thus the new objectwill become a copy of the predecessor modified by the differences copiedinto the CAS from the copying source pool.

At steps 1506, 1508, the Data Mover 502 pushes the data into the CAS.The data is accompanied by an object reference and an offset, which isthe target location for the data. The data may be sparse, as only thedifferences from the predecessor need to be moved into the new object.At this point the incoming data is broken into depth-0 chunks sizedsmall enough that each can be represented by a single depth-0 hash.

At step 1510, the data hash module generates a hash for each depth-0chunk.

At step 1512, read the predecessor hash at the same offset. If the hashof the data matches the hash of the predecessor at the same offset, thenno data needs to be stored and the depth-1 and higher objects do notneed to be updated for this depth-0 chunk. In this case, return toaccept the next depth-0 chunk of data. This achieves temporaldeduplication without having to do expensive global lookups. Even thoughthe source system is ideally sending only the differences from the datathat has previously been stored in the CAS, this check may be necessaryif the source system is performing differencing at a different level ofgranularity, or if the data is marked as changed but has been changedback to its previously-stored value. Differencing may be performed at adifferent level of granularity if, for example, the source system is asnapshot pool which creates deltas on a 32 KiB boundary and the CASstore creates hashes on 4 KiB chunks.

If a match is not found, the data may be hashed and stored. Data iswritten starting at the provided offset and ending once the new data hasbeen exhausted. Once the data has been stored, at step 1516, if theoffset is still contained within the same depth-1 object, then depth-1,depth-2 and all higher objects 1518 are updated, generating new hashesat each level, and the depth-0, depth-1 and all higher objects arestored at step 1514 to a local cache.

However, at step 1520, if the amount of data to be stored exceeds thedepth-1 chunk size and the offset is to be contained in a new depth-1object, the current depth-1 must be flushed to the store, unless it isdetermined to be stored there already. First look it up in the globalindex 1116. If it is found there, remove the depth-1 and all associateddepth-0 objects from the local cache and proceed with the new chunk1522.

At step 1524, as a quick check to avoid visiting the global index, foreach depth-0, depth-1 and higher object in the local cache, lookup itshash in the local store established in 1502. Discard any that match.

At step 1526, for each depth-0, depth-1 and higher object in the localcache, lookup its hash in the global index 1116. Discard any that match.This ensures that data is deduplicated globally.

At step 1528: store all remaining content from the local cache into thepersistent store, then continue to process the new chunk.

Reading an object out of the CAS is a simpler process and is commonacross many implementations of CAS. The handle for the object is mappedto a persistent data object via the global index, and the offsetrequired is read from within this persistent data. In some cases it maybe necessary to recurse through several depths in the object handletree.

CAS Object Network Replication

As described under FIG. 11, the Replicator 1110 is a service toduplicate data objects between two different content addressable stores.The process of replication could be achieved through reading out of onestore and writing back into another, but this architecture allows moreefficient replication over a limited bandwidth connection such as alocal- or wide-area network.

A replicating system operating on each CAS store uses the differenceengine service described above together with the temporal relationshipstructure as described in FIG. 13, and additionally stores on aper-object basis in the temporal data structure used by the CAS store arecord of what remote store the object has been replicated to. Thisprovides definitive knowledge of object presence at a certain datastore.

Using the temporal data structure, it is possible for the system todetermine which objects exist on which data stores. This information isleveraged by the Data Mover and Difference Engine to determine a minimalsubset of data to be sent over the network during a copy operation tobring a target data store up to date. For example, if data object O hasbeen copied at time T3 from a server in Boston to a remote server inSeattle, Protection Catalog Store 908 will store that object O at timeT3 exists both in Boston and Seattle. At time T5, during a subsequentcopy from Boston to Seattle, the temporal data structure will beconsulted to determine the previous state of object O in Seattle thatshould be used for differencing on the source server in Boston. TheBoston server will then take the difference of T5 and T3, and send thatdifference to the Seattle server.

The process to replicate an object A is then as follows: Identify anobject A0 that is recorded as having already been replicated to thetarget store and a near neighbor of A in the local store. If no suchobject A0 exists then send A to the remote store and record it locallyas having been sent. To send a local object to the remote store, atypical method as embodied here is: send all the hashes and offsets ofdata chunks within the object; query the remote store as to which hashesrepresent data that is not present remotely; send the required data tothe remote store (sending the data and hashes is implemented in thisembodiment by encapsulating them in a TCP data stream).

Conversely, if A0 is identified, then run the difference engine toidentify data chunks that are in A but not in A0. This should be asuperset of the data that needs to be sent to the remote store. Sendhashes and offsets for chunks that are in A but not in A0. Query theremote store as to which hashes represent data that is not presentremotely; send the required data to the remote store.

Sample Deployment Architecture

FIG. 16 shows the software and hardware components that comprise oneembodiment of the Data Management Virtualization (DMV) system. Thesoftware that comprises the system executes as three distributedcomponents:

The Host Agent software 1602 a, 1602 b, 1602 c implements some of theapplication-specific module described above. It executes on the sameservers 1610 a, 1610 b, 1610 c as the application whose data is undermanagement.

The DMV server software 1604 a, 1604 b implements the remainder of thesystem as described here. It runs on a set of Linux servers 1612, 1614that also provide highly available virtualized storage services.

The system is controlled by Management Client software 1606 that runs ona desktop or laptop computer 1620.

These software components communicate with one another via networkconnections over an IP network 1628. Data Management Virtualizationsystems communicate with one another between primary site 1622 and datareplication (DR) site 1624 over an IP network such as a public internetbackbone.

The DMV systems at primary and DR sites access one or more SAN storagesystems 1616, 1618 via a fibre-channel network 1626. The servers runningprimary applications access the storage virtualized by the DMV systemsaccess the storage via fibre-channel over the fibre-channel network, oriSCSI over the IP network. The DMV system at the remote DR site runs aparallel instance of DMV server software 1604 c on Linux server 1628.Linux server 1628 may also be an Amazon Web Services EC2 instance orother similar cloud computational resource.

VSS Requestor and VSS Provider in a Single Process Space

VSS (Volume Shadow Copy Service) is a framework that exists on versionsof Microsoft Windows operating systems since 2003. This frameworkfacilitates the cooperation among backup products, applications andstorage components to create application-consistent backups. However,the VSS framework anticipates that each component will perform specifictask independently, which can lead to unnecessary inefficiencies andoverhead, as will be explained further below.

FIG. 17 is a schematic diagram of the VSS framework on a MicrosoftWindows operating system. The VSS framework includes the Volume ShadowCopy Service 1701, the VSS requestor 1702, the VSS writers 1703, and theVSS provider 1704.

The Volume Shadow Copy Service 1701 coordinates communication betweenvarious VSS Framework components such as the VSS Requestor 1702, the VSSWriter 1703 and the VSS Provider 1704 and enables creation ofapplication consistent snapshot. The Volume Shadow Copy Service Service1701 is, for example, part of Microsoft Windows Operating System and isprovided by Microsoft. The Volume Shadow Copy Service 1701 provides thesystem infrastructure for running VSS applications on Windows-basedsystems. The Volume Shadow Copy Service 1701 can be largely transparentto the user and developer. In some embodiments, the Volume Shadow CopyService 1701 is configured to perform a number of different tasks, suchas coordinating activities of providers (e.g., VSS Provider 1704),writers (e.g., VSS Writer 1703), and requesters (e.g., VSS Requestor1702) in the creation and use of shadow copies (e.g., shadow copies area snapshot of a volume that duplicates all of the data that is held onthat volume at one well-defined instant in time); furnish the defaultsystem provider; and implement low-level driver functionality necessaryfor any provider to work

Backup vendors develop VSS Requestor 1702. The VSS Requestor is a backupprogram or agent that may initiate backup operations. Typically, VSSRequestors are installed on the system that needs to be backed up andrun as a separate process. A VSS Requester can be any application thatuses the VSS API (e.g., the IVssBackupComponents interface) to requestthe services of the Volume Shadow Copy Service 1701 to create and manageshadow copies and shadow copy sets of one or more volumes. Anillustrative example of a requester is a VSS-aware backup/restoreapplication, which uses shadow-copied data as a stable source for itsbackup operations.

Applications developed by Microsoft and other vendors (e.g., SQL,Oracle, Microsoft Exchange applications) come with VSS Writers 1703 thatare specific to the product that have the ability to freeze theapplication and make the application store on disk self-consistent andrecoverable. Each VSS writer is built specifically for an applicationand is typically installed along with the application. For example, SQLServer VSS Writer coordinates I/O operations with VSS Service for SQLServer. The VSS writer freezes and thaws application I/O operations whenrequested by the VSS Service to allow VSS Providers to captureapplication consistent snapshot of the application data store. If nowriters are present during a VSS backup operation, a shadow copy canstill be created.

Storage technology vendors develop VSS Provider 1704, which is capableof capturing the state of the self-consistent image of the applicationat the moment the application is frozen by the VSS Requestor 1702, sothat application can resume normal operation. The VSS Provider 1704takes some sort of snapshot, e.g. either within the software on thesystem, or using hardware and/or software external to the system. TheVSS Provider is installed on the system where application runs andtypically runs as an independent process. As an illustrative example, inresponse to a request from a requester, a provider generates events tosignal applications of a coming shadow copy, and then creates andmaintains that copy until it is no longer needed. While a shadow copy isin existence, the provider can create an environment where there areeffectively two independent copies of any volume that has been shadowcopied: one the running disk being used and updated as normal, the othera copy that is disk fixed and stable for backup. A default provider canbe supplied as part of the Windows operating system.

In the conventional use of the VSS Framework, the Requestor and Providerare independent processes, and do not communicate with each otherdirectly. They are designed to be general purpose, and operate withother Providers and Requestors respectively. When a VSS requestor makesa request to the VSS service, the VSS service blocks the VSS requestorthread until it receives a response from the VSS provider. Further theVSS framework does not provide the VSS writer with any context about theVSS requestor that caused the VSS service to invoke the VSS provider.

The VSS application programming interface contains no means for the VSSRequestor and VSS Provider to communicate with each other, other thanfor the Requestor to learn of the existence of various Providers and toselect one of them. All VSS Requestor and Provider actions arecoordinated by VSS Service.

The techniques described herein provide for a VSS requestor and VSSprovider that are implemented as separate threads as part of the sameprogram. Therefore, for example, when the VSS service blocks the VSSrequestor thread that called API to create snapshot, the VSS provider isstill executing as part of the same program and can therefore useintra-process communication means to communicate with the VSS requestor.For example, the VSS provider can determine what is to be created (e.g.,what kind of copy, for which program, etc.), how much storage space touse, where the storage space is to be allocated from, and/or otherinformation that is not otherwise available to a VSS provider. In someexamples, the mechanism used for communication between the VSS requesterand the VSS provider is a callback handler, which provides forintra-process communication. For example, the VSS Provider communicateswith the VSS Requestor using the callback handler and notifies the VSSRequestor when specific events occur such as Snapshot Commit Event.

The VSS provider can also be created as a stealth provider so that itdoes not show up as a provider for other VSS requestors. For example,the VSS Requestor can register the custom VSS Provider with the VSSframework when it starts a backup operation using VSS framework API andunregister the VSS Provider once the backup operation is complete. Thiscan make the VSS Provider available on the system only during the backupoperation.

There are benefits that can be realized by having the Requestor andProvider communicate with each other. For example, the Requestor canprime the Provider with configuration information from the backup jobthat it is about to run, which may be used by the Provider to choose theresource pool or other parameters in the creation of the snapshot. Otherexamples of functionality that may require communications messagesinclude the coordination of snapshots across multiple hosts, thesignaling at the exact moment of consistency to the external backupserver, or many other use cases.

In the present disclosure, approaches are presented for communicationbetween the Requestor and Provider and realize the benefits discussedabove.

FIG. 18A illustrates a combined VSS requestor and VSS provider, inaccordance with some embodiments. As shown in FIG. 17, there is theVolume Shadow Copy Service 1701, and the VSS Writers 1703. In thisembodiment, the VSS Requestor 1802 and the VSS Provider 1804 have beenlinked together as separate sets of threads in the same single processspace 1805. This enables the Requestor and the Provider to use one ormore of several intra-process communication channels 1806, such asglobal memory, pointer passing or thread signaling to synchronize andexchange information between the Requestor and Provider threads.

The custom VSS Provider 1804 can be a software or hardware snapshotprovider. It can implement COM interfaces such asIVssSoftwareSnapshotProvider, IVssProviderCreateSnapshotSet, etc.prescribed by VSS framework. VSS Service 1701 invokes the custom VSSProvider using these interfaces. In addition, the VSS Provider 1804 ismade aware of the presence of VSS Requestor 1802 by registering acallback handler with the VSS Provider 1804. The callback handler isregistered by the VSS Requestor 1802 and it acts as a communicationchannel between the VSS Requestor 1802 and the VSS Provider 1804. TheVSS Provider notifies the VSS Requestor when certain events occur usingthe callback handler.

The VSS Requestor 1802 invokes VSS framework functionality using VSSFramework API such as IVssBackupComponents interface. The VSS Requestor1802 registers a callback handler with the custom VSS Provider toreceive notifications from the Provider and processes callbacknotification messages received from the Provider.

This embodiment does not preclude the Requestor or the Provider fromfunctioning in their conventional roles. It enables additionalfunctionality that can provide more efficient and effective solutions todata protection and data recovery problems.

FIG. 18B illustrates an intra-process communication scheme between theVSS Requestor 1802 and the VSS Provider 1804 during an exemplary backupsequence. In this illustration, a single Connector process 1844 hostsboth the VSS Provider and VSS Requestor. Connector 1844 is a backupprogram developed by backup vendor for backing up applications. It runsas a single process and all the resources consumed by the VSS Requestorand VSS Provider are owned by that process.

At step 1812, the Connector registers the custom VSS Provider 1804 withVSS framework when the connector starts running at step 1810. Theconnector starts listening for backup requests once it is fully up andrunning.

At step 1814, once a request to backup an application is received fromthe data management virtualization system, the connector starts thebackup sequence at step 1816 using the VSS Requestor 1802. At step 1816,the VSS Requestor 1802 checks if the application is running andavailable for backup and the writer for the each of the applicationbeing backed up is in healthy state

At step 1818, if the storage used by the application is exported fromthe Data Management Virtualization System (“DMV”), the VSS Requestor1802 selects the custom VSS Provider 1804 for backup sequence 1820,primes the VSS Provider 1804 with application specific information andregisters a callback handler with the VSS Provider. For example, the VSSRequestor 1802 can select the VSS provider using the VSS API (e.g.,IVssBackupComponents::AddToSnapshotSet). The determination to use thecustom VSS Provider is made by comparing the LUN (Logical Unit Number)id of storage volume used the application with the LUN id of volumesexported by DMV.

For each application that needs to be backed up, the VSS Requestor 1802selects the volumes used by that application's data store for VSSsnapshot. After selecting volumes for snapshot, the VSS Requestor 1802requests creation of snapshot (e.g., by using the IVssBackupComponentsinterface) at step 1824. The request to create a snapshot is made usinga separate thread, as the VSS Service will block the calling threaduntil the snapshot either succeeds or fails. This allows the VSSRequestor 1802 to continue to receive callback messages using thecallback handler previously registered from the VSS Provider 1804 whilethe snapshot creation is still in progress. As part of snapshotcreation, the VSS Service 1701 requests VSS Writer 1803 to freeze theapplications that are being backed up. Once the applications are frozen,the VSS Service 1701 requests the VSS Provider 1804 to create snapshotof volumes used by application. In response to the request to createsnapshot, the VSS Provider notifies the VSS Requestor 1802 using apreviously registered callback handler that VSS framework is ready forcreating snapshot and suspends itself at step 1826.

Upon receiving notification from the VSS Provider at step 1826, the VSSRequestor 1802 requests the DMV to create a snapshot of volume used bythe application data store and waits for response at step 1828. DMV thencreates a copy-on-write snapshot of requested volumes at step 1830 andresponds to the VSS Requestor 1802 with the status of request at step1832. After the snapshot is successfully created, the VSS Requestor 1802notifies the VSS Provider 1804 to resume VSS processing so thatapplication can resume normal processing at step 1834. For example, theapplication is un-frozen by the VSS Service after the snapshot creationis completed successfully or when the snapshot creation fails.

It is required for the entire processing between step 1824 and step 1834to be completed within a certain timeframe (e.g., 10 seconds) otherwisethe VSS Writer 1803 rejects the application freeze requests. Having boththe VSS Requestor and the VSS Provider within the same process canreduce the communication overheads associated with inter-processcommunication and helps complete snapshot processing within the timeinterval.

VSS Requestor 1802 completes the backup at step 1836 and notifies theDMV the status of backup request at step 1838.

The Connector unregisters the VSS Provider 1804 at step 1840, and theConnector stops running at step 1842.

In some embodiments, the Requestor and Provider are not within the sameprocess space. For example, the Requestor and Provider can bemulti-threaded within their own process spaces, and communicate witheach other through a side channel using an inter-process communicationschannel such as shared memory, sockets or even disk based files.

SmartCopy Protection for Out-Of-Band Data

This disclosure describes an extension to the Virtual Data Pipeline(VDP) technology to cover protection of data that is on internal ornetworked drives. Out-of-band (OOB) refers to the fact that the storageis not presented to the host by the VDP system. For example, theout-of-band storage can be storage that cannot be physically accessed bythe VDP system, such as a local drive (e.g., “C” drive) of adesktop/laptop or network storage provided from a private network. Thestorage is visible to the host through some other path, not provided foruse by the VDP system in a way that the VDP system can directly accessthe storage. Such out-of-band storage can be, for example, data storedin a file system (e.g., a Window's file system, Linux file system,etc.).

Prior to this disclosure, there was no means to use the VDP technology,which is located from the host system to be backed up, to capture andprotect data that resided on out-of-band drives that cannot be accesseddirectly by the VDP system, such as direct attached drives within a hostsystem (e.g., within a laptop or a desktop), or Network attached storageserved by an external File Server with its own storage (e.g., in aprivate network, such as a company network).

With the present disclosure, it becomes possible to protect the datawith all of the benefits of the VDP system. Images are capturedincrementally, and yet are available as virtual full images. The data istemporally organized, making it easier to capture time ordereddependencies and to derive benefits in smaller storage space, moreeffective deduplication and more efficient data management. The virtualfull images also enable easy presentation to hosts without a need tolayer incremental upon incremental upon full, reducing the time torestore or clone, and enabling the mount capability.

The data can be protected using a smart copy agent that executes on thehost to be protected. The smart copy agent can be presented with storagefrom the VDP system and use the presented storage to copy the data thatis only visible to the host. For subsequent copies after the first copy,the smart copy agent can copy only the new data that has changed sincethe last copy. For example, the smart copy agent can compares the datawith the copy to update only data that has changed. For example, ratherthan using timestamps, the smart copy agent can walk the file structureto see if there are any new files, if any of the metadata for a file haschanged (e.g., which indicates the file has been modified since the lastcopy), or if any files were deleted since the last copy.

FIG. 19A is a schematic diagram of a system 1900 providing out-of-bandprotection, in accordance with some embodiments. The system 1900includes the host 1901, which runs the Smartcopy agent 1902. Thesmartcopy agent 1902 is a process that runs in the background on thehost 1901, which is described in further detail herein. The system 1900also includes primary database storage 1903 in communication with theHost 1901, which cannot be directly accessed by the VDP system 1904. Thedata to be protected lives on direct attached or out-of-band ornetworked attached storage 1903. The Smartcopy agent 1902 is incommunication with the VDP system 1904. The VDP system 1904 is incommunication with performance pool 1906, which is directly accessibleto the VDP system 1904. The system 1900 also includes backup stagingvolumes 1905 (e.g., also referred to herein as a “staging disk”) thatare allocated from the performance pool 1906 such that the performancepool 1906 is in communication with the host 1901, the Smartcopy agent1902, and the performance pool 1906.

Referring to the host 1901, this can be, for example, a computer orvirtual machine running a Microsoft Windows, Linux, AIX, Solaris, orHP-UX operating system that is supported by Smartcopy agent 1902. Thehost 1901 can be connected to the VDP 1904 via a network connection(e.g., and optionally via fibre channel).

Referring to the Smartcopy agent 1902, this can be, for example, thesmartcopy program executable for the specific operating system runningon host 1901. This program can be installed on the host using standardpackage installation procedures for the specific operating system. Forexample, on Windows the user runs a setup program that installs theSmartcopy agent to run as a service. As another example, on a Linux hostthe user installs an RPM package that installs the Smartcopy agent as adaemon process. In some embodiments, the Smartcopy agent runscontinually in the background and communicates with the VDP system usingTCP/IP.

Referring to the Primary Storage 1903, this can be, for example, a disksystem that is usable by host 1901, such as an installed SATA, SCSI, orSAS hard disk, or a SAN-provided disk that is attached by fibre channelor other high-speed disk interconnects, such as a NetApp or EMC SANdevice. Protected host 1901 uses the primary storage 1903 to read andwrite files on a file system.

Referring to the VDP System 1904, this can be, for example, the VDPsystem described herein.

Referring to the Backup Staging Volumes 1905, this can be, for example,a virtual disk device that is provisioned from free space available inPerformance pool 1906.

Referring to the Performance pool 1906, this can be, for example, a disksystem that is attached to VDP system 1904 via fibre channel, such as aNetApp or EMC SAN device.

FIG. 19B describes message and data flows of a system providingout-of-band protection. The VDP system 1904 activates the protectionpolicy 1907 (e.g., according to a SLA). In step 1908 the VDP system thensends a backup request to agent 1902. Smartcopy agent 1902 receives thebackup request in step 1909. Smartcopy agent 1902 then sends a stagingdisk requirement request back to the VDP system in step 1910. VDP System1904 receives the response from the Smartcopy agent 1902. VDP System1904 then presents a staging disk to protected host 1902. Smartcopyagent 1902 mounts the presented staging disk in step 1913. Smartcopyagent 1902 then copies, deletes, or updates files on staging disk 1905so that the contents match primary storage 1903 in step 1914. In step1915 Smartcopy agent 1902 sends the results of the backup to VDP System1904. VDP System 1904 receives the backup results in step 1916 andunmaps the staging disk from protected host 1901. If the backup wassuccessful, then VDP System 1904 takes a point-in-time snapshot of thestaging disk 1905. VDP System 1904 then catalogs the backup metadata.

Referring to step 1907, this is a protection policy being activated.This can be started by, for example, a user manually running aprotection policy, or it was run as part of a schedule defined for theprotection policy. This protection policy can be previously created by auser to protect a specific piece of data on the protected host 1901, inthis case primary storage 1903.

Referring to step 1908, the VDP System 1904 sends a request, such as astring containing XML describing which volume should be backed up, toSmartcopy agent 1902.

Referring to step 1909, the Smartcopy agent 1902 receives the request,for example a string containing XML describing which volume to be backedup. The volume to be backed up is identified using operating systemspecific names. For example, on Windows the volume can be referred to asC:, D:, E:, etc. On Linux the volume can be referred to as /, /usr,/mnt/volume1, etc. If sent using XML, for example, the XML can alsocontain any other data required to back up the volume, such ascredentials for authenticating on the host, options specified by theuser to control other backup features. A job identifier can also beincluded in the XML so that the job can be tracked by the VDP. This XMLis parsed and the results are used in step 1910.

Referring to step 1910, the Smartcopy agent 1902 calculates the size ofthe staging disk that is required for this backup. For example, if therequest is to back up a volume that is 40 gigabytes in size, it willdetermine that the staging disk must be at least 40 gigabytes in size aswell. This requirement is sent to the VDP System 1904 as a stringcontaining XML describing the requirement.

Referring to step 1911, the VDP System 1904 receives the responsecontaining the required size for the staging disk (e.g., the XMLresponse string). The VDP System finds any existing staging disk for theprotection policy that is currently running. If the existing stagingdisk is at least as large as the required staging disk size, theexisting staging disk can be used as the staging disk. If no existingstaging disk was found (e.g. this is the first backup for thisprotection policy or previous staging disks have been expired) or theexisting staging disk is smaller than the required size, a new stagingdisk can be allocated from the Performance Pool 1906. Once a stagingdisk has been allocated for a backup, that same staging disk can be usedfor all subsequent backups of the same protection policy, unless thesize of the disk must be increased, in which case a larger staging diskis created and used for future backups. If a user expires all backupsfor a protection policy, the staging disk is deleted, and a new stagingdisk must be allocated for subsequent backups of the same protectionpolicy. The point-in-time snapshots in step 1917 depend on the stagingdisk they were created from, but they are not used as staging disks, andchanges to the staging disk do not affect the point-in-time snapshot,nor do changes to a point-in-time snapshot change the contents of thestaging disk on which they depend.

Referring to step 1912, the VDP system 1904 presents the staging disk tothe protected host 1901. This can be done for example, via iSCSI, orfibre channel if 1901 is a physical computer. For example, the stagingdisk can be presented using standard techniques for making a diskvisible over iSCSI or fibre channel. The VDP System is the target andthe protected host is the initiator using SCSI terminology. In someembodiments, if 1902 is a virtual machine, then the disk is firstpresented to the virtual machine hypervisor, such as VMware ESXi, andthen the staging disk is added to the virtual machine 1901. In someembodiments, the VDP System sends an XML string to the Smartcopy agentcontaining the LUN identifier of the staging disk which is used in step1913.

Referring to step 1913, the Smartcopy agent 1902 scans its storage busto find the iSCSI or fibre channel disk for a physical machine. If 1901is a virtual machine, the disk will appear as a SCSI disk presented bythe hypervisor and the SCSI bus is scanned to find the staging disk. TheSmartcopy agent continues to scan the bus until it finds a disk with thesame LUN identifier that the VDP System sent in step 1912. Once thestaging disk has been found on the storage bus, it is partitioned andformatted if the staging disk is not already formatted. The disk isformatted with a file system that is the same as primary storage 1903.If the primary storage 1903's file system cannot be determined or is notsupported, the staging disk is formatted with the standard file systemfor the type of operating system that protected host 1901 runs. Forexample, Microsoft Windows systems can use NTFS and Linux can use ext3.The staging disk is then mounted at a mount point on the protected host1901. For example, on a Linux system it can be mounted under a directorylocated at /act/mnt, and the specific directory can be named based onthe current job identifier and time. As another example, on a Windowssystem, it can be mounted under a directory located at C:\Windows\act,and the specific directory can be named based on the current jobidentifier and time.

Referring to step 1914, the Smartcopy agent 1902 will copy any files ordirectories from the primary storage 1903 to the staging volume 1905 ifthe file exists on the primary storage and not on the staging volume.Any files or directories that do not exist on the primary storage butexist on the staging volume will be deleted from the staging volume. Insome embodiments, any files or directories that have different contentor metadata, such as timestamps, file attributes, or securitydescriptors, will be updated on the staging volume to match the primarystorage. When this step is complete, the staging volume will be areplica of the primary storage. The only differences, if any, may be aresult of filesystem incompatibilities or file system metadata that isspecific to the disk, such as the volume identifier. The staging volumeis unmounted from the host after the copying and deleting havecompleted.

Referring to step 1915, the result of the backup is sent from theSmartcopy agent 192 to the VDP System 1904. This is a string containingXML describing the results of the backup, such as whether or not it wassuccessful, and if it was not successful, the error code describing theerror that occurred.

Referring to step 1916, the VDP System 1904 receives the backup result,which is a string containing XML. This is parsed to determine if thebackup was successful or not. The staging disk is then unmapped from theprotected host 1901.

Referring to step 1917, this step is only reached if the result of step1916 indicates that the backup was successful. A point-in-time snapshotof the staging disk is created using the VDP Systems' flashcopy feature.This snapshot of the staging disk is a virtual full copy of thefilesystem on the staging disk and is stored in the Performance Pool1906. It has the same characteristics of an in-band backup within theVDP System. These snapshots can be mounted as fully independent disks,they can be cloned, restored, duplicated for long-term storage, ortransported across a WAN for disaster recovery and business continuity.

Referring to step 1918, the new point-in-time snapshot, if any, iscataloged as the most recent backup of the host filesystem. When thenext time the same filesystem on the same protected host 1901 needs tobe protected, the staging disk will be reused, substantially reducingthe amount of data that must be copied by the Smartcopy agent in futurebackups.

Smart Copy for Database Backup

This disclosure describes extensions to the Virtual Data Pipeline (VDP)system to support protection and replication of database systems thatare not otherwise accessible to the VDP system. For example, theextensions allow the VDP system protect a database that the VDP systemcannot communicate with directly (e.g., the database is stored on alocal drive of the system, or it is stored in network storage providedby a private network that is not accessible to the VDP system).

With the embodiment described in this disclosure, relational databasesystems (RDBMS) such as Oracle, SQL Server and Sybase can be protectedand replicated with all of the benefits of the workflow of the VirtualData Pipeline system. Data capture may be done efficiently andincrementally, and the backups may be virtual full backups, which can bemounted, cloned and restored quickly and efficiently.

Prior to this disclosure, there was no means to use the Virtual DataPipeline (VDP) technology, which is located on a different host from thehost system to be backed up, to capture and protect data that resided inrelational databases on direct attached drives within a host system, ornetwork attached storage served up by an external file server with itsown storage. There is no way for the VDP system to communicate with suchdatabases directly to manage the databases; the database is visible tothe host through some other path.

With the present disclosure, it becomes possible to protect the databasedata with all of the benefits of the VDP system. Backup images arecaptured incrementally, and yet are available as virtual full images.The data is temporally organized, making it easier to capture timeordered dependencies and to derive benefits in smaller storage space,more effective deduplication and more efficient data management. Thevirtual full images also enable easy presentation to hosts without aneed to layer incremental copies upon incremental copies upon fullcopies (and so on), reducing the time to restore or clone, and enablingthe instant mount capability.

The data can be protected using a RDBMS agent that executes on the hostto be protected. The RDBMS agent can be presented with storage from theVDP system and use the presented storage to copy the database, which isonly visible to the host. The RDBMS agent can be configured to use adatabase copy tool such that the database copy can be loaded and used bya database system, even after subsequent incremental copies.

FIG. 20A depicts a configuration 2000 for protection of a database inaccordance with some embodiments. The configuration 2000 includes thehost 2001, which runs the RDBMS-enabled agent 2002. The RDBMS agent 2002is a process that runs in the background on the host 2001, which isdescribed in further detail herein. The configuration 2000 also includesprimary database storage 2003 mounted on the host 2001, which cannot bedirectly accessed by the VDP system 2004. The database data to beprotected lives on direct attached or out-of-band or networked attachedstorage 2003. The RDBMS agent 2002 is in communication with the VDPsystem 2004. The VDP system 2004 is in communication with performancepool 2006, which is directly accessible to the VDP system 2004. Theconfiguration 2000 also includes backup staging volumes 2005 that aremounted on the host 2001 during backup and allocated from theperformance pool 2006, and in communication with the RDBMS agent 2002;the staging volumes are created out of the performance pool 2006.

The host 2001 is a server with running RDBMS, which is the actualdatabase that needs backup protection. The RDBMS running on the host2001 uses primary database storage which is considered Out-of-Band tothe VDP System 2004. For example, an Oracle database runs on Linuxserver with database storage supplied from an array other than theperformance pool 2006. This database server lives in a data center andserves as persistent data repository for various applications.

The RDBMS agent 2002 is a software component running on the host 2001.The RDMBS agent 2002 is deployed on the host 2001 and communicates withVDP System 2004, primary database storage 2003 and backup stagingvolumes 2005 during database backup. The RDBMS agent 2002 communicateswith RDBMS and utilizes available conventional method for incrementalforever methodology.

The primary database storage 2003 is data storage of RDBMS running onthe host 2001. The primary database storage 2003 can either be locallyattached disk drives or network attached storage consumed by RDBMSrunning on the host 2001.

The VDP system 2004 can be an embodiment of the Virtualization DataProtection system described herein.

The backup staging volumes 2005 is a backup destination provisioned outof the performance pool 2006 and managed by VDP system 2004. The backupstaging volumes 2005 is mounted to the host 2001, thus allowing read andwrite operations to be performed. The RDBMS agent 2002 writes RDBMSbackup artifacts onto the backup staging volume 2005.

The performance pool 2006 is a storage pool used by the VDP system 2004to perform protection operations. Protection operation requests storagefrom the performance pool 2006 to be used for backups, for example thebackup staging volume 2005 is provisioned out of the performance pool2006 and mapped to the host 2001.

FIG. 20B illustrates an exemplary process of configuring a database foruse with the VDP system. At step 2050, the backup request is sent fromthe VDP system 2004 to the RDBMS agent 2002 for backup staging volume2005 requirements. Step 2050 is explained in further detail below withrespect to steps 2007-2012 of FIG. 20B. At step 2051 the VDP system 2004processes the backup staging volume 2005 requirements message andprepares the backup staging volumes 2005. Step 2051 is explained infurther detail below with respect to steps 2013-2015 of FIG. 20B. Atstep 2052 the VDP system 2004 maps the backup staging volumes 2005 tothe host 2001; the RDBMS agent 2002 makes the backup staging volumes2005 ready to be receiving backup I/Os. Step 2052 is explained infurther detail below with respect to step 2016-2017 of FIG. 20B. At step2053 the RDBMS agent 2002 performs backup of the primary databasestorage 2003. Step 2053 is explained in further detail below withrespect to steps 2018-2020 of FIG. 20B. At step 2054 the RDBMS agent2002 merges the incremental changes with last backup to make it up todate and copies required database artifacts onto the backup stagevolumes 2005. Step 2054 is explained in further detail below withrespect to steps 2021-2024 of FIG. 20B. At step 2055 the VDP system 2004creates a snapshot of the backup staging volumes 2005 and catalog themetadata. Step 2055 is explained in further detail below with respect tosteps 2025-2027 of FIG. 20B.

FIG. 20C illustrates an exemplary detailed message and data flow of theincremental-forever backup protection. FIG. 20A, FIG. 20B detailcommunications between VDP System 2004 and RDBMS Agent 2002, andexecution steps of VDP System 2004 and RDBMS Agent 2002.

A protection policy is activated in step 2007 by scheduler of the VDPsystem 2004. A protection policy is part of SLA (service levelagreement) which is defined by end user and stored by the VDP system2004 and applied to RDBMS on the Host to be protected 2001. SLA has aschedule defined and being evaluated by scheduler of VDP system 2004.The policy is activated by the schedule once it is determined that thedefined criteria are met.

At step 2008 the VDP system 2004 sends requests to the RDBMS agent 2002on the protected host 2001. The requests of step 2008 consist ofinstructions for backup operations to be consumed by the RDBMS agent2002.

At step 2009, after receiving a backup instruction from the VDP system2004, the RDBMS agent 2002 processes the backup instructions.

At step 2010, the RDBMS agent 2002 communicates with RDBMS to determinethe configured size of the RDBMS to be used as the size for the backupstaging volume 2005. In some examples, the end user has option to use auser specified size to override the calculated size. The size of thebackup staging volume 2005 is calculated in such a way to allow theincremental backup to execute forever without running out of space forbackups.

At step 2011, the RDBMS agent 2002 sends the size requirement and uniquesignature for the backup staging volume 2005 to the VDP system 2004. Theunique signature is a string that can be used to uniquely identify abackup staging volume in the performance storage pool 2006. For example,for an Oracle database, its SID (Oracle System ID) can be used as theunique signature for the backup staging volume 2005.

At step 2012, the VDP system 2004 received the size requirement andunique signature for the backup staging volume 2005 from the RDBMS agent2002.

At step 2013, the VDP system 2004 checks existing staging disks in theperformance pool 2006 to determine whether a disk with this uniquesignature and the required size already exists. The method moves to step2014 if the staging disk is not found, or forward to step 2015 ifstaging disk is found in the performance pool 2006.

At step 2014, the VDP system 2004 creates a backup staging disk withrequired size and signature from the performance pool 2006.

At step 2015, the VDP system 2004 retrieves the backup staging diskfound in step 2013 from the performance pool 2006.

At step 2016, the VDP system 2004 presents the staging disks eithercreated in step 2014 or retrieved in step 2015 as the backup stagingvolumes 2005 to the Host to be protected 2001. The presentation is tomap the staging disk to the Host to be protected 2001, an unique disksignature is sent to the Host to be protected 2001.

The RDBMS agent scans the buses to find the presented backup stagingvolumes 2005, and if they are uninitialized volumes, formats them andcreates a file system to receive the data in step 2017. Any file systemtype that is natively supported on the host operating system, and iscompatible with the database software is acceptable. In the preferredembodiment, the NTFS file system is used for Windows systems, and theext3 file system is used for Linux systems.

At step 2018, the RDBMS agent 2002 first determines whether a fullingest is required for backup of the RDBMS by examining the backupstaging volume 2005. The RDBMS agent 2002 will move to step 2020 ifprevious backup artifacts are found and it is determined no full ingestis needed. Otherwise the RDBMS agent 2002 will move to step 2018 for afull ingest.

At step 2019, the RDBMS agent 2002 copies the logical contents of thedatabase to be protected onto the staging volumes 2005. The purpose ofthis copy is to create an image of the database data in a format inwhich it can be started up on a similar host machine. The contents ofthe captured image of the database are an exact copy of the originaldatabase at a particular point in time.

The methodology for creating the image copy may vary from one databaseapplication to another. It will be clear to someone skilled in the arthow to create an image of the particular database system using thedatabase vendor's conventional methodology. For example, vendors oftenprovide backup tools for database systems that allow the database to becopied in a manner such that it can be loaded as an operationaldatabase. For example, for an Oracle database, the preferred method ofcreating the image is to use the Oracle RMAN command “Backup incrementallevel 0 as copy database with tag ‘xyz’”. Otherwise, using conventionalbackup methodologies that are not designed for the database may notpreserve the database structure, and therefore a database backed upusing conventional backup methodologies may not result in an operationaldatabase (e.g., the backed up database cannot be loaded and used by thedatabase utilities).

At step 2020, the RDBMS agent 2002 performs an incremental backup of theprimary database storage 2003, writes backup artifacts onto the backupstaging volume 2005.

At step 2021, the RDBMS agent 2002 merges the changes with the imagecopies of data files to make these image copies up to date on the backupstaging volumes 2005.

Once the image creation on the staging disk is complete, the RDBMS agentmay copy additional artifacts to the staging disk as required to makethe backup image self-consistent in step 2022. A person skilled in theart may recognize backup control files, archive log files, databaseconfiguration files, and VSS writer metadata documents as artifacts thatmay be copied on to backup staging volumes 2005.

In step 2023, the RDBMS Agent 2002 unmounts the staging volumes 2005 toprevent the image copies and other backup artifacts being overwritten ordamaged to keep the backup data integrity.

At step 2024, the RDBMS agent 2002 sends backup result messagescontaining metadata of the backup to the VDP system 2004 to finalize thebackup operation. Now the involvement of the RDBMS agent 2002 iscompleted.

At step 2025 the VDP system 2004 unmaps the backup staging volumes 2005from the host to be protected 2001. End user has choice to keep thebackup staging volumes 2005 mapped to the host 2001 to override defaultbehavior.

At step 2026 the VDP system 2004 creates a point in time snapshot of thebackup staging volumes 2005. This step is to create point-in-time flashcopy of the backup staging volumes 2005. The flash copy can be purposedfor multiple uses with the data contents having the state at the timethe snapshot was taken. For example, a flash copy image of the backupstaging volumes 2005 can be mounted to a host in quality assurancedepartment for testing.

At step 2027 the snapshot of the backup staging volumes 2005 iscataloged as the most recent backup of the RDBMS application.

The next time the protection policy is activated on schedule to the samedatabase under protection, the same image mode disks representingstaging volumes 2005 may be reused, reducing the amount of data movementthat may potentially be required. In this situation, as done in step2015, the VDP system 2004 presents the same backup staging volumes 2005to the host 2001. The RDBMS agent 2002 scans these volumes and mountsthe file system, making the previously created image visible to the hostin step 2017. The RDBMS Agent 2002 now updates the image on the stagingdisks, bringing it up to a more recent point in time in steps 2020 and2021. Once again, the detailed methodology may vary according to thedatabase vendor's conventional methodology. For an Oracle database, asan example, the procedure is to run the RMAN command “backup incrementallevel 1 for recover of copy with tag ‘xyz’ database”, followed by thecommand “recover copy of database with tag ‘xyz’”. In step 2022 is onceagain to copy artifacts for self-consistency to the staging volumebefore unmount the staging volumes in step 2023 and send messages to VDPSystem 2004. VDP system 2004 unmaps the staging disks from the host2001, and creates a latest point in time snapshot of the staging volumesin step 2026. This set of snapshots is cataloged in step 2027 as themost recent backup of the database application.

The snapshots of the staging disks are virtual full backups of thedatabase under protection and have all of the characteristics of in-bandbackups within the VDP system. These backup images can be mounted andstarted as full independent instances; they can be cloned or restored,deduplicated for long term storage, or transported across the WAN fordisaster recovery and business continuance.

When the copy is performed from the RDBMS device, the initial copy isperformed on the host 2001. The copy operation is performed by the RDBMSAgent 2002. The copy is done to a set of volumes 2005 that is presentedby the VDP system 2004.

Quick-Linking Services in the User Interface

This disclosure relates to user interfaces for products in the DataManagement Virtualization space. More specifically, it describes thesystems and methods for Quick-linking across data management, dataprotection, disaster recovery and business continuity services runningon top of storage management stack from the data management provider asdescribed herein. Quick links can be defined for actions that spanacross a number of different services defined by the service objects bysharing data and information across the services using a shared servicescache. The Quick-linking service can be used to automatically performsteps in a workflow based on context information for subsystems of asystem implementation (e.g., data management, data protection, disasterrecovery and business continuity services), whereas without the contextinformation the steps would need to be performed manually (e.g., by asystem user). For example, the roles and rights of the user areinherently represented in the views and functionality available within aservice is scoped accordingly. Quick-linking can resolve the problem ofcontext awareness and user initiated job identification whilesimplifying the end-to-end workflow. Access to the underlying subsystemsof the data management virtualization display can allow the datamanagement virtualization display to use the persona of the particulardata management virtualization implementation, coupled with data andother information that can be gleaned from the underlying subsystems, toautomatically perform steps of jobs that would otherwise need to bemanually controlled and manipulated by the user.

Data Management Provider Desktop

FIG. 21 depicts the Data Management Provider Desktop in accordance withsome embodiments. The Data Management Provider Desktop is the userinterface that implements systems and methods for the Data ManagementVirtualization Engine 306. The Data Management Provider Desktop allowsusers to manage, monitor, configure and report on managed data,applications and their associated storage as viewable from the DataManagement Virtualization Engine 306. Within the Data ManagementProvider Desktop, each of the primary use cases is broken down into agroup of actions that a user persona may be interested in performingwith the system. Each persona based logical grouping constitutes a‘Service.’

In some embodiments, the supported services can include a Dashboard. TheDashboard can include, but is not limited to, aggregate views of currentand historical data and performance for the system components. It canalso include point of time state of the major components that make upthe Data Management Virtualization Engine, any critical events requiringimmediate user intervention.

In some embodiments, the supported services can include a DomainManager. Domain Manager can include a group of functions, which aretraditionally performed by a user responsible for ConfigurationManagement, Host Management and Security Management within theirorganization.

In some embodiments, the supported services can include a Service LevelAgreement (SLA) Architect. This allows a user to create and manage SLAsthat specify the business requirements of an application for its datalifecycle. As described further herein, a SLA is a detailedspecification that captures the detailed business requirements relatedto the creation, retention and deletion of copies of the applicationdata.

In some embodiments, the supported services can include an ApplicationManager, which is described in further detail referring to FIG. 23.

In some embodiments, the supported services can include a ReportManager. A Report Manager provides for basic reporting and businessanalytics based on data integration and connectivity with the underlyingplatform subsystem.

In some embodiments, the supported services can include a SystemMonitor, which is described in further detail with respect to FIG. 24.

The Data Management Provider Desktop 2100 runs on a client PC (e.g.,running Windows or a Mac OS), and communicates over IP Network 2101 withthe Data Management Virtualization Engine 306 (e.g., described inreference to FIG. 3). In some embodiments, the communications use secureHTTPS protocol. In some embodiments, the API is REST based. The DataManagement Virtualization Engine is connected to Hosts 2103 and StorageResources 2102 via either IP or fiber channel (FC) networks 2104. TheHost 2103 can be, for example, a physical or virtual machine whereprograms, applications, and file systems of interest reside. The StorageResources 20102 can be the locations where actual data is stored.

FIG. 22 depicts a schematic diagram of a service manager interfacewithin the Data Management Provider Desktop UI with quick links, inaccordance with some embodiments. The service manager runs within theDesktop UI. A group of services 2201 resides within the window manager2200. Each of these services the dashboard 2202, the domain manager2203, the SLA architect 2204, the application manager 2205 and thesystem monitor 2206 is connected to each other via Quick links 2208.Each service resides on top of a shared cache 2207, is managed by the UIController 2209 and follows the Model View Controller architecture. DataModels 2210 hold objects for the services in question. Quick links inaddition to providing an entry point to these services also allow forsharing data between services

Referring to the window manager 2200, the window manager can be a shellinside which the views and services are built and visualized. Windowmanager abstracts the underlying OS underneath and provides the facilityto work with the windowing framework (e.g. resizing, styling, andlayouts).

Referring to the quick links 2208, the quick links 2208 tie the services2201 together to seamlessly guide the user through their user cases. Forexample, the quick-linking mechanism can be used for various jobinitiating user actions related to backup data management, including butnot limited to mount, clone, restore, unmount, unmount and delete,expire, failover, test failover, syncback, and/or the like. The quicklinks 2208 provide a visual and intuitive approach of switching contextbetween the services 2201 while providing users with real time statusand feedback on their existing subsystems.

Quick link is a process by which the services can communicate with eachother, update their context, and ultimately resolve use cases tied tocustomer needs. Quick links are implemented using the facilities andcapabilities of the Service Object; they can be invoked and managed byany combination of services. The underlying API on the platform plays animportant part with the process of switching the context, it providesfacilities for lookups and complex quires based on the quick link usecases.

Referring to the shared services cache 2207, this can be a datastructure that holds data in name/value pairs. These data sets areaccessible to all the services via public methods. There are multipleways of implementing the shared services cache, such as, for example, ahashmap.

Referring to the UI controller 2209, the UI controller 2209 can beimplemented similar to a model view controller (MVC) based architecturewhere the business knowledge resides in the UI controller 2209 andseparates the UI views in the Window Manager 2200 from the data model2210. The UI controller 2209 can be configured to listen to changes inboth the UI views and/or the data models.

Referring to the data model 2210, this is where most of the servicecentric data is stored. Each of the services can be configured with itsown data model and the controller gets the events from the views,converting them into entities to update the model.

FIG. 22A shows the implementation and architecture of the service andtheir linkages. Service Object 2222 implements a window manager 2220 andseveral targeted views 2224. It also has pointers to the shared servicescache 2207. The UI controller 2209 exercises most of the functionalitywithin the service object.

According to the systems and methods disclosed herein, the various viewsfor the data management virtualization system are grouped togetherlogically into a set of services 2201. The architecture provides theshared services cache 2207 to allow all of the services to access,transport, and save data across the services 2201. The architecture alsoprovides service objects 2222 that can be used to define the componentsof a particular service (e.g., including a particular window manager forthe service, the targeted views for the service, how the serviceinteracts with the shared cache, and methods for the controller tomanage the service). The service object 2222 includes a name for theservice object, a reference to a window manager 2220, an indication ofthe currently loaded view for the service object 2222, a set of accessmethods to the shared services cache 2207, public access methods forother service objects to use to access the service object, and a datamodel (e.g., data model 2210).

The window manager 2220 can be implemented for a particular serviceobject 2222. As described herein, the window manager is a shell insidewhich the views are built and visualized to the user. The window manager2220 includes a name for the window manager 2220, a window managerinstance, a current load state, a current visual state, and publicaccess methods for the window manager 2220.

The targeted views 2224 are the views for the service object 2222. Eachtargeted view 2224 includes a name for the targeted view 2224, an HTMLtemplate for the targeted view 2224, and public access methods for thetargeted view 2224.

The UI controller 2209 is configured to manage the service objects 2222.In some embodiments, the UI controller 2209 manages data transfer amongthe service objects 2222 via the shared services cache 2207. Asdescribed above, the shared services caches 2207 can be used to sharedata in arrays of key/value pairs. The data can be shared, for example,for functional context, visual context, and/or the like.

Jobs

In Data Management domain there are numerous user-initiated operationsthat can affect the state of the system, these often require carefulmonitoring and reporting. Each one of these user driven end-to-end taskstriggers a system level activity (e.g., a process in the Data ManagementVirtualization Engine 306), which henceforth is referred to as a ‘Job.’A job may include multiple sub jobs with their own state and lifecycle.Managing and monitoring of jobs within the Data ManagementVirtualization subsystem is a significant activity and performed fromwithin the System Monitor service.

In some embodiments, a Service Policy Engine is responsible for managingjobs. The Data Management Virtualization Engine 306 performs severaljobs, such as backup, recovery, etc. that span multiple storage pools.If, for example, there are multiple volumes that the operation relatesto, then each job instantiates other sub-jobs.

Services 2201

Referring to the services 2201, each service is implemented as anindependent application within its own window manager. Each serviceshares a framework of libraries and code written for handling commonfunctional behaviors and visual components. An array of service objectsis stored within the base window manager and is instantiated when thedesktop application is launched.

Application Manager 2205

FIG. 23 depicts an exemplary Application Manager Service. TheApplication Manager Service provides a graphical user interface fordiscovering, protecting and managing application lifecycle. Allapplications protected by Service Level Agreements (SLAs) are displayedand organized logically inside its framework. Backup, restore and otherData Management Virtualization Engine 306 operations are initiated fromthis service.

A visualization of the Application Manager service 2301, which includesa navigation listing 2302 for elements of Application manager whichincludes all the applications under management by the Data ManagementVirtualization Subsystem. A list of backups 2303 for the selectedapplication, an image ID 2304 is used to identify the images and 2305are the supported actions on the image. The image is a sample visual ofa service and where the user could trigger an action to initiate a quicklink.

System Monitor 2206

FIG. 24 depicts an exemplary System Monitor service, which in the DataManagement Provider Desktop is responsible for handling all user visibleactivities related to jobs, including monitoring and management. FIG. 24also identifies the layout of System Monitor.

The System Monitor service 2401 lists the jobs and events in thesubsystem. Jobs listing 2402 lets the user select the context for thedata grids. Filters 2403 allows one to narrow down the data set in thegrid based on pre-determined filters. Individual jobs 2404 are listed ina grid, each with a corresponding Job ID 2405.

FIG. 25 depicts the display of specific details of a particular job fromwithin a System Monitor service. A user can choose an individual job byeither double clicking a row or choosing ‘View Details’ option brings upthe details as displayed in FIG. 25.

Service Manager

Service Manager is a framework that allows for managing of individualservice instances. It provides for several service related operationsincluding adding new service, remove existing service, displaying andhiding of services and managing their visual states.

Quick-Linking

As described above, the quick links 2208 tie the services 2201 togetherto seamlessly to guide the user through their user cases. To illustratethese techniques a sample task of backup mount operation is used. Theservices used for a backup mount operation include the ApplicationManager 2205 and System Monitor 2206. A user initiates the mountoperation from within the Application Manager 2205. The systemidentifies the application and its associated backup that may bemounted, takes in the input from user regarding the mount point andindividual backup volumes desired to be mounted. Once user inputs havebeen validated, the Data Management Virtualization Engine 306 systeminitiates a mount request which instantiates a corresponding job and/orrequired sub-jobs.

An association between jobs is maintained within the platform subsystemof Data Management Virtualization Engine 306. The request to instantiatethe job and/or required sub-jobs is sent from the Data ManagementProvider Desktop 2100 to the Data Management Virtualization Engine 306.On successful instantiation of the job, the Data ManagementVirtualization Engine 306 returns the unique ID of the job (or parentjob when there are required sub-jobs) back to the Data ManagementProvider Desktop 2100. In some examples, the request can be transmittedto the Data Management Virtualization Engine 306 via a Web Serviceapplication programming interface (API). The Web Service can be a Javabased service custom implementation of representational state transfer(REST) architectural principals. The communication between platform andthe user interface can use JavaScript Object Notation (JSON)-formattedstrings over the Hypertext Transfer Protocol-Secure (HTTPS) protocol.Data Management Provider Desktop 2100 then uses this ID to walk throughthe list of all available jobs.

FIG. 25 depicts the display of specific details of a particular job fromwithin a System Monitor service 2206, in accordance with someembodiments. Once a match with a job is identified, the job details arerequested from the Data Management Virtualization Engine 306. Thesedetails are appropriately formatted, visualized and made available tothe user in the appropriate service. In this example, the servicecontext is switched to System Monitor 2206 and the Job Details window ispresented to the user. If the job in is running state, the details ofthe job are dynamically updated. The status information is retrievedfrom a query initiated by the Data Management Provider Desktop 2100 aspart of a client-side polling loop every 5 seconds. The poll starts assoon as the job platform confirms that a job has been successfullylaunched.

In graphical context, the Application Manager 2205 view (FIG. 23)switches to the Job Details view (FIG. 25) and the background serviceswitches to the System Monitor 2206 view (FIG. 24).

Service Context Switching

An array of currently loaded services is maintained on a Data ManagementProvider Desktop 2100 instance level and it also stores the state ofeach service, including whether it is user visible. When a switchingcontext occurs, it updates the stored states in the array of loadedservices and uses the Window Manager 2200 to change the visual context.It also involves instantiating the service instance if one has not beenloaded previously (e.g., is not in the array of currently loadedservices).

FIG. 26 describes the user flow without Quick-linking. At step 2601, theuser initiates a job from the Application Manager 2205. At step 2602,the system triggers a job or list of jobs in response to user request.At step 2603, the System Monitor 2206 organizes jobs by start date,status and job type. At step 2604, the System Monitor 2206 displays anordered list of jobs, which periodically updates based on new userrequests. At step 2605, the user switches service from ApplicationManager 2205 to System Monitor 2206. At step 2606, the user chooses theappropriate filters to narrow down the list of possible jobs in SystemMonitor 2206. At step 2607, the user identifies the job based on actiontype in System Monitor 2206. At step 2608, the user selects the job anddouble clicks on the selected row to get job specific details in SystemMonitor 2206. This process may require several complex steps.

FIG. 27 describes the user flow with Quick-linking. The user initiates ajob from the Application Manager 2205 at step 2701. In step 2702, theData Management Provider Desktop 2100 displays the details forappropriate job using quick linking. The process is complete withoutadditional interaction from the user.

Smart Data Synchronization

The following section deals with operational data within the datamanagement system. As data is stored within the system, metadataregarding the time of snapshots, content-addressable handles, and othersuch metadata accumulates in the system. This data is called operationaldata. Operational data also includes policies, schedules and systemconfigurations. The platform server is a centralized data managementsystem that collects and maintains a copy of operational data from eachsub-system locally for each sub-system that the platform server ismanaging. As well, remote site replication requires one system for eachsite. If the two sites are close, a single management console may beused for both.

In some situations, the use of multiple data management systems mayafford advantages. This may occur, for instance, when the amount of userdata stored in the system exceeds a maximum threshold, such as 127terabytes (TB). The use of multiple data management systems may befacilitated by the use of a central management server that synchronizesoperational data for each of the multiple data management systems.Approaches for replicating and synchronizing operational data arediscussed below. In some embodiments, a different synchronizationstrategy can be deployed for data based on the data itself. In someembodiments, a different synchronization strategy can be deployed fordata based on the number of data records (e.g., a small number ofrecords, a medium size set of records, or a large size set of records).Synchronizing operational data can, for example, eliminate the need fora user of the central management server to go to each data managementsystem to manage each system; instead they can simply manage all datamanagement systems through the central management server. Users caneasily get a global view of all data management systems they areresponsible for with the help of a central management server.

Traditionally, operational data is synchronized by comparing data fromsource and target, adding data to target that exists only in source,deleting data that only exists in target, and update target data withdata from source if they are different. Techniques are disclosed hereinto replicate operational data. Different techniques can be used based onthe number of operational data records. For example, a small set cansimply be replaced each time synchronization occurs; as the data oftenchanges and can be done quickly. For a medium set, both timestamps andrecord IDs can be used to synchronize the data (e.g., since the numberof IDs is manageable, and can be used to indicate deletion information.For a large set, record IDs alone can be used to synchronize the data inconjunction with a tolerance number to account for a simultaneousprocessing window (e.g., since some operations cannot be guaranteed tooccur prior to other operations). This is possible because large sets ofdata typically do not change once they are created, and is typicallydeleted based on some retention policy.

FIG. 28 depicts a Management Console, 2801, that can manage multipleData Movement Virtualization Engines, 306, in accordance with someembodiments. This depicts a scenario when data on multiple Data MovementVirtualization Engines 306 is synchronized to a single ManagementConsole 2801. Management console 2801 may include a database ofenterprise manager operational data 2802, which includes a replicatedcopy of the data management operational data 3001 for each of themultiple Data Movement Virtualization Engines 306 synchronized to theManagement Console 2801. These Data Management Operational Data 3001maintain regular operation, or are generated through normal operations.They are typically stored in a Relational Database System, althoughother types of storage are also possible.

FIG. 29 depicts a database of Enterprise Manager Operational Data 2802,in accordance with some embodiments. This database can store, forexample, operational data related to data required to performoperations, and the results of these operations. The operational datastored in the database include service level agreement (SLA) data 2902,Protection data 2904, History data 2905, Event Store data 2903 andApplication data. A unique ID within a Data Movement VirtualizationEngine can be used to uniquely identify each record stored in thedatabase.

For Management Console 2801 to manage multiple Data MovementVirtualization Engines 306, these operational data may be synchronizedto the Management Console 2801, which also resides in a relationaldatabase. Each Data Movement Virtualization Engine 306 is alsoassociated with a unique ID. The combination of the unique record ID andData Movement Virtualization Engine ID uniquely identifies a record inthe Management Console, and the origin of the records.

FIG. 29 depicts examples of Management Console Operational Data 2802,which contains data that are replicated from Data MovementVirtualization Engines. FIG. 30A depicts examples of Data ManagementOperational Data 3001, in accordance with some embodiments. These datamay include SLA data 3002, Protection data 3004, and History data 3005.Event Store data 3006 and Application data 3306 may also be co-locatedwith the operational data 3001. Application data represent applicationsthat the data management system manages. SLA data represent the policiesthat are used to protect the applications. Protection data representpolicies that are used to protect various applications. History datarepresent all protection operations performed on the system, whether itis successful or not. Event data collect all events occurred on the datamanagement system.

In general Operational Data can be divided into small-size sets of data,typically less than a few hundred records, such as SLA 3002; ormedium-size sets of data, usually in the thousands, such as ProtectionData 3004; or large-size sets of data, which can be into hundreds ofthousands or even more, such as History Data 3005. In some embodiments,the size of a record does not matter, but rather just the number ofrecords. For the large-size sets of data, as they are generated in sucha high rate, they are typically not modified, and are only deleted inbulk. History Data 3005 fit this criterion, as they represent historicalrecords of all of the operations performed in Data ManagementOperational Data 3001. Another example is Event Data 3003.

Synchronization of Small Sets of Data (or Frequently Changing Sets ofData)

For tables (sets of records) that contain small sets of records,traditional comparison of all records is sufficient. This involvescopying all data from source to target and comparing them. As the datasets is small, network bandwidth consumption and CPU to process them istypically be minimal. A small set of data can include less than 1,000operational data records.

Synchronization of Medium-Size Sets of Data

As the total size of a set of records increases, the cost of fetchingall records and comparing them increases also. A different strategy canbe used to more efficiently synchronize the data. A medium-size set ofdata can include over 1,000 operational data records but less than100,000 operational data records.

In some embodiments, the current timestamp when each time a record ischanged (e.g., an update or a create) is included. As each recordcontains a last modified date, it is possible to examine only recordsthat have changed since the last synchronization. Each synchronizationrequest is accompanied with a last synchronization time, so only newchanges are sent from source to target.

Since saving changes in a relational database does not happeninstantaneously, the use of last synchronization time may not be enoughto distinguish between two simultaneously-saved changes. As well,records may be missed if the timestamp of a record falls right betweenthe assigned timestamp and synchronization time. This problem can beavoided by adding (or rolling back) a tolerance adjustment to thesynchronization time. This may result in more data being sent, but mayguarantee correctness, as each record is still compared and updated ifchanges is made, or discarded if no changes are made. The tolerance canbe calculated, as maximum duration required committing changes.

There is often a lag between the time a timestamp is set on a record andthe time it is actually saved, such that if a first record was savedbefore a second record, concurrent transactions running in the systemmay result in time stamping the second record before the first record.The tolerance can be calculated to be more than enough to cover thatperiod, and more. While a larger tolerance results in more records to beexamined, it can better guarantee correctness. By examining concurrenttransactions allowed and maximum delay due to actual saving of the data,a reasonable tolerance can be chosen. Then an additional factor can beadded for safety (e.g., doubling the calculated tolerance does notaffect the performance too much, but ensures data is properlysynchronized). A tolerance of 2 minutes, for example, can be used as theadditional factor.

This synchronization procedure works well if data are never deleted. Inthe case of records that can be deleted, however, there is no place tostore the timestamp within the record, as the record no longer exists.To account for data being deleted from the source, a different strategycan be applied. As each record is tagged with a unique ID, in additionto sending last synchronization time, all known IDs in the target systemcan be sent to the source side. Source side can then examine the sourceIDs, and send back a list of IDs that is no longer in the source side.Target side can then delete the data that are tagged by the source to bedeleted.

FIG. 30B depicts Protection Data 3004 in Data Management OperationalData 3001, which has Protection Data records with ID 10002 that wasmodified on January 1 (1-1; the year is not necessary for this example),ID 10004 that was modified on January 13 (1-13), ID 10012 that wasmodified on January 12 (1-12), ID 10015 that was modified on January 16(1-16), and ID 10020 that was modified on January 20 (1-20). Theellipses for each entry in Protection Data 3004 indicate that otherfields can be included, as well as the data for the record itself.During previous synchronization, ID 10002 that was modified on January 1(1-1), ID 10004 that was modified on January 13 (1-13), ID 10007 thatwas modified on January 2 (1-2), ID 10012 that was modified on January12 (1-12), and ID 10015 that was modified on January 12 (1-12) weresynchronized to Protection Data 2904 in Management Console 2801. In thisexample, the last synchronization of Protection Data 3004 to ProtectionData 20904 happened on January 15 (1-15), so the 1-16 modification torecord ID 10015, and the 1-20 modification to record ID 10020 inProtection Data 3004 has not been synchronized to Management Console2801 (Protection Data 2904), and record 10007 was deleted fromProtection Data 3004, which is not reflected in Management Console 2801.The last synchronization time (e.g., 1-15 for this example) is recordedfor easy calculation for next synchronization.

During the next synchronization, Management Console 2801 can send arequest to Data Management Virtualization Engine 306 to synchronizeProtection Data, as shown in FIG. 30C. The request 30041 containstimestamp from which the system subtracts a tolerance (we will use 1 dayfor this example), as described above. So in Request 30041, a requestfor all data from 1-14 onwards is sent, instead of 1-15, aftersubtracting the tolerance of one day, by Management Console 2801 to DataManagement Virtualization Engine 306. As another example, the timestampscan include minute information, second information, etc. For example, ifthe tolerance is 60 seconds, if the timestamp of the last backup was1-15 at 1:00 pm, then Management Console 2801 would request data from1-15 at 12:59 pm from the Data Management Virtualization Engine 306.

When Data Management Virtualization Engine 306 receives the request30041, the Data Management Virtualization Engine 306 retrievesProtection Data that have been created/modified since 1-14, namely ID10015 modified at 1-16, ID 10020 modified at 1-20. Contents for ID 10015(1-16) and ID 10020 (1-20) are sent back to the management console 2801,with all known IDs in Protection Data 3004 (ID 10002, ID 10004, ID10012, ID 10015, ID 10020), are sent back to Management Console 2801 inreply 30042, shown in FIG. 30D. Management Console 2801 can just updateID 10015, and create ID 10020. Record 10007 is then deleted as it nolonger exists in the known IDs list in the reply 30042, which means itis deleted from the Protection Data 3004 since the last synchronizationto Protection Data 2904. This process can be repeated for each of theData Management Virtualization Engines 306 that the Management Console2802 needs to manage (e.g., each with its own modified data and list ofIDs for each Data Management Virtualization Engine 306).

Synchronization of Large Sets of Data (Data that does not Change at all,Only Bulk Delete Old Data)

The use of last modified time incurs the cost of retrieving current timeand tagging a record with the time. As each record is tagged with aunique ID, by using an ID generation strategy, it can further improvethe efficiency of synchronizing large sets of data, if the data is notmodified after it is created. A large set of data can be, for example, aset with over 100,000 operational data records.

One strategy is to assign ID in an ever-increasing manner, and the sameID is not reused. These IDs may have gaps in their order, but in generallarger IDs indicate that the record is created later. With this systemor method, there is no need to tag each record with a creation time.

To efficiently assign ever-increasing ID numbers, the following strategycan be employed. A chunk of IDs are reserved to be dished out (e.g.,1024 IDs, 2048 IDs, etc.), and the last possible assigned ID is recordedin the database (e.g., ID 1024, ID 2048, etc.). Each time an ID isneeded, one is used. When all IDs are used, another chunk of IDs arereserved, and the largest ID possible is again recorded. If the systemcrashes, those that are not given out can be discarded, and the processcan start from the recorded largest ID again. This way the ID can beassigned efficiently, and the ID is in general in the same order as timesuch that a larger ID is generally assigned to data that occurs later intime. Using a predetermined size, such as 64 bits, for an ID canguarantee that the ID does not loop around.

During synchronization, the largest ID in the target system (andsubtracting some tolerance number as in the case of timestamp) is sentto the source system, instead of the synchronization timestamp. Onlyrecords with IDs that is larger are sent to the target system. Inaddition, those known IDs within the tolerance are also sent from thetarget to the source. So the source only needs to send those that arelarger than the ID sent from target and not in the list of IDs that areduring the tolerance period. Using this system or method, a targetsystem can take records sent from source as is, without having tocompare records.

FIG. 30E depicts History Data 3005 in Data Management Operational Data3001, with records with IDs 1, 2, and so on up to 10010, 10011, 10012,10013, 10014, and again up to 10100. Each ID is associated with a job(e.g., ID 1 is associated with Job_00001). The ellipses for each entryin Protection Data 3004 indicate that other fields can be included, aswell as the data for the record itself. During a previoussynchronization of the history data 3005 to the history data 2905, IDs1, 2, . . . , 10010, 10011, and 10013 are already synchronized toHistory Data 2905 in Management Console 2801.

During the next synchronization, Management Console 2801 sends arequest, 30051 to Data Management Virtualization Engine 306 tosynchronize History Data, shown in FIG. 30F. To generate the request,the Management Console 2801 first examines the largest ID that it owns,which is 10013. The Management Console 2801 then subtracts a tolerancenumber of records from the largest ID. In this example, we can use atolerance number of 3. Counting backwards for 3 records from 10013results in an ID of 10010. So the request 30051 is for all history datathat is larger than 10010. As another example, a tolerance of 200 can beused, such that counting backwards for 200 records from the largest ID10013 record results in ID which may be ID 9801. The tolerance can bechosen by examining the concurrent transactions and maximum delays incommitting a transaction, with some factor for safety.

The Management Console 2801 also determines if it already has anyrecords between the calculated ID 10010 and the largest ID 1014. TheManagement Console 2801 determines that History Data 2905 includes IDs10011 and 10013, and therefore it does not need to receive another copyof this data. So the Management Console 2801 generates request 30051 forall data greater than ID 10010, but do not include (10011, 10013). Therequest 30051 is then sent to Data Management Virtualization Engine 306.

FIG. 30G depicts Data Management Virtualization Engine 306 afterreceiving the request 3051; it retrieves data with IDs larger than10010, but excludes data with IDs 10011 and 10013. The Data ManagementVirtualization Engine 306 sends the results back to Management Console2801 in reply 3052. Management Console 2801 can then just add theHistory Data as reply 30052 will only contain data that the History Data2905 does not already have. This process can be repeated for each of theData Management Virtualization Engines 306 that the Management Console2801 needs to manage (e.g., each with its own modified data and list ofIDs for each Data Management Virtualization Engine 306).

As described above, different strategies may be applied to differenttypes of data in the system, depends on the properties of the data. Asdata accumulate in the system, the properties of the data may be knownin advance, and different strategies can be applied to achieve increasedperformance based on these properties.

Synchronization of data from Data Movement Virtualization Engine 306, toManagement Console 2800, may be accomplished through a combination ofthese strategies. Examples of such combinations include:

-   -   SLA, 2902, on Data Movement Virtualization Engine, 306, is        synchronized to SLA, 3002, on Management Console, 2800, with the        small sets of data strategy.    -   Protection, 2904, on Data Movement Virtualization Engine, 306,        is synchronized to Protection, 3004, on Management Console,        2800, with the medium sets of data strategy.    -   History, 2905, on Data Movement Virtualization Engine, 306, is        synchronized to History, 3005, on Management Console, 2800, with        the large sets of data strategy.        Location-Based Hash Index Caching

The disclosed data storage and deduplication engine converts anarbitrarily-sized computer file or disk image source to one or morefixed sized blocks of data. These blocks of data may be written into acapacity-optimized storage pool (CAS), as described above at step 1528.As data is read into the CAS, data blocks are written, or persisted, tothe file system in batches in the order they were received. Each batchis preceded by a persist header block. The persist header containsinformation about each data block that follows it, including size,location, and hash code. A batch persist operation contains about twothousand data blocks from only one source at a time.

The systems and methods described herein improve the performance ofreading data from a deduplicated data store. Reading data from adeduplicated data store is often time consuming because by its verynature data stored in a deduplicated data store is often spread outthrough the data store by various pointers (and/or the like) to avoidduplicate data. Therefore, reading from the deduplicated data storerequires reading data spread throughout the data store (e.g., ratherthan sequentially, such as reading a single file stored on disk). Forexample, each hash for the data must first be located, then used to lookup an index of where the associated data is stored on disk, and then thedata can be read from the system (e.g., which often requires many timeconsuming mechanical movements of hard drives). The hash information isoften stored using a B-tree, which is a data structure that keeps datasorted and allows searches, sequential access, insertions, and deletionsin logarithmic time. However, using B-trees is often not fast enoughwhen reading from a deduplicated data store.

The systems and computerized methods described herein provide for acustom persist header data structure that is used to store new data asit is written to the deduplication data store. The persist headerincludes a set of hashes for the data represented by the persist header.The systems and computerized methods also provide for a degrading hashtable (also referred to as a “scoreboard”) that is used to cacherecently-accessed hashes, as well as hashes that are near therecently-accessed hashes. For example, if a hash is read for data storedin a persist header, the remaining hashes associated with the persistheader can be pre-cached into memory to improve the speed of the read(e.g., because there is a high likelihood that data after the looked uphash will also be read, since it was written during a same writeoperation to the deduplicated data store). If the next data request isfor a hash pre-loaded in the degrading hash table, the pre-loaded dataavoids needing to look up the data in the master hash index (which canbe time consuming).

FIG. 31 is a schematic diagram of the persist header and subsequent dataon disk in accordance with some embodiments. The persist header 3100precedes a number of data blocks 0, 1, 2, 3, 4 . . . 1919 to be writtento disk as a logically single operation. For the persist header 3100,blocks 0-1919 represent the first blocks 0-1919 from the source image3102. Persist header 3104 also precedes 1920 blocks (0, 1, 2, 3, 4 . . .1919). For the persist header 3104, blocks 0-1919 represent the second1920 blocks, or blocks 1920-3839 from the source image 3102. As data isstored, a persist header is written to disk first and then followed byup to 1920 individual data bocks. The persist header maintains theidentifying hash information about each of the ensuing data blocks.

While the data stored in each persist header is shown as correspondinglinearly to data stored on the source image, this is for illustrativepurposes only. For example, for deduplicated data storage, only new data(and not duplicate data) is written from the source image to thededuplicated data store. Therefore, in these embodiments the persistheader only stores the new data for each write, so the data stored byeach persist header may not necessarily correspond to a linearrelationship with the data stored in the source image. But by arrangingthe data in persist headers in this manner, the persist header storespotentially related content, nearby content, and/or the like.

FIG. 32 is a schematic diagram of the persist header data structure. Thearray of 1920 hashes 3205 contains the hash value of each of thesucceeding blocks of data following the persist header as documented inFIG. 31. The array of 256 chunks 3204 contains information about theexact location, format, and validation checksum of a “chunk” of up to 16hash values.

I/O Header 3201 is a data structure (e.g., C Language structure) that isshared for all system metadata stored on disk. This structure caninclude, for example, data for error checking, a page identificationnumber, a page version, a timestamp of when it was written, and a datatype identifier.

Chunk Count 3202 is an integer that contains the number of chunks ofuser data that follow this persist header. A “chunk” contains up tosixteen 4K (4096 bytes) of data.

Index count 3203 is an integer that contains the number of hashes thatare contained in the subsequent previously mentioned chunks that followthe persist header.

Array of 256 Chunks 3204 is an array of structures (e.g., C Languagestructures) that describe the subsequent chunks of user data including,for example: location, compression, check sum, number of hashes (up to16), and encryption.

Array of 1920 Hashes 3205 is an array of structures (e.g., C Languagestructures) that contain the SHA1 hash values for all the 4K data blocksdescribed by this persist header.

Unused space 3206 is a filler to 64K (65536 bytes) to align the wholepersist header structure.

Checksum 3207 is an integer checksum of the structure to be used toverify data integrity.

FIG. 33 is an exemplary schematic diagram of a deduplication hash index.For example, the deduplication has index can be implemented in a B-tree.Interior pages form an indexed array of references to leaf pages whichin turn are an indexed array reference to the hash values contained inthe persist header (FIG. 32). A portion of the hash value is used as anindex into the interior pages. In a hierarchical fashion, more of thehash value is then used to identify the appropriate leaf page. Lastly,the persist header is used to find the relevant data block.

Referring to “Interior Pages,” 3301 this is an array of structures(e.g., C Language structures) that is stored on disk and describes thelocation of block of references to “Leaf Pages.” 3302 Each interior pagestructure contains, for example, an array of disk locations in which tofind leaf pages.

Referring to “Leaf Pages,” 3302 this a structure (e.g., C Languagestructure) that is stored on disk and cross references hash values to“Persist Headers.”3200

Referring to “Persist Headers,” see FIG. 32.

The hash index is a mechanism by which the data stored in the system isfound based on its hash value. The three levels shown in FIG. 32(interior page 3301, leaf page 3302, and persist header 3200) create athree tier hierarchy which provides the ability to locate any specificdatum by its hash without requiring the entirety of the index to belocated in system RAM. For example, the interior pages can reside in RAMwhile the leaf pages are loaded and unloaded as needed. Similarly,persist headers are loaded as needed as referenced by leaf pages.

FIG. 34 is a schematic diagram of the “page cache” in accordance withsome embodiments. The page cache is an array of pre-allocated memorypages in the system; each memory page is referred to as a “cache page.”The cache pages are managed within the “page cache” as an LRU (LeastRecently Used) list. The page cache can be used as the centralrepository for memory within the system. For example, most dynamicmemory used by the application can be obtained from the page cache.

When specific datum is required, the page cache LRU list is searched ina linear fashion from most recently used to least and, if found, theappropriate cache page 3403 is removed from the LRU list and returned.If the datum is not found, the least recently used cache page isreturned. When a cache page is released, it is placed in the front ofthe LRU list so that it may be found quickly if needed again.

Referring to “Cache Page Array” 3401, this is an array of Cache PageArray Entry 3402 structures (e.g., C Language structures) that describea number of pre-allocated 64K (65536 bytes) memory blocks. All cachepages are sized to be 64K (65536 bytes). Interior pages 3301, leaf pages3302, persist headers 3200 as well as all internal structures within thesystem are sized to fit into one cache page.

Referring to “Cache Page Array Entry” 3402 this is a structure (e.g., CLanguage structure) that describes a single cache page entry in thecache page array. Where “list” 3411 is a structure (e.g., C Languagestructure) for managing linked list inclusion (e.g., this is used tomanage the LRU). The “Index” 3412 is an index value. The “flags” 3413value is used to describe how the page is being used within theapplication, for instance, the flags may indicate whether or not thecache page is in use or whether or not it contains data that needs to besaved. The “tran_id” 3414 field is used to identify the current taskusing the cache page. The “hash code” 3415 field is a C languagestructure that typically contains a SHA1 hash code for the cache pageidentified by this entry. The cache page 3403 may be used for any numberof purposes; interior page 3301, leaf page 3302, persist header 3200, orother system data. The hash identifier is used find a specific cachepage in the page cache. The “data” 3416 field is a pointer (e.g., CLanguage pointer) to the memory described by this entry. In someembodiments, the data points to a persist header as shown in FIG. 32.

FIG. 35 is a schematic diagram of the degenerating scoreboard system.The “Hash Table Array” 3501 is an array of data structures which containa pair of numbers: an index into the “Persist Header Reference” 3502array and an index into a persist header “array of 1920 hashes” (FIG.3205). The persist header reference is an MRU (most recently used) arrayof persist header hash codes. The persist header hash codes are used toretrieve a “persist header” (FIG. 32) from the “page cache” (FIG. 34).Data is found by using a portion of its identifying hash value as anindex into the “Hash Table Array” 3501 which results in a persist headerreference index and an index into the persist header “array of 1920hashes” 3205. The page header reference index is used to obtain thepersist header (FIG. 32) hash code from the “persist header reference”3502. The hash code 3512 is then used to retrieve the persist headerdata from the page cache (FIG. 34). The index to the persist header“array of 1920 hashes” 3204 is used to locate the specific identifyinghash code.

Referring to “Hash Table Array” 3501 this is an array of structures(e.g., C Language structures) that link a hash value to an entry in the“Persist Header Reference” 3502. Each entry contains three fields: “HashFragment” 3521, “Page Index” 3522, and “Hash Index” 3523. “HashFragment” is a part of a SHA1 that is used to check that the entry foundmatches the hash value referenced. “Page Index” is a numerical indexinto the “Persist Header Reference.” “Hash Index” is an index into a“Persist Header” “array of 1920 hashes” (FIG. 32).

Referring to the “Persist Header Reference” 3502 this is an array ofstructures (e.g., C Language structures) that reference “PersistHeaders” (FIG. 32) as contained within the “Page Cache” (FIG. 34). Eachentry in the Persist Header Reference includes a reference to a cachepage entry 3402, a hash code, and a disk reference. The cache page 3511is used as an index into the cache page array (see FIG. 3401). The hashcode is used to verify the proper cache page array entry was identifiedwhen searching the cache page array for the cache page.

The hash table array is used as a degrading hash table (or“scoreboard”). For example, as data is read from a deduplicated datastore, the first hash is retrieved (e.g., as described in FIG. 36, suchas in a B-tree), and the array of 1920 hashes 3205 from the persistheader 3200 that includes the first hash is loaded into the hash tablearray. If a subsequent request is for data with a hash stored in thehash table array, then the request can be processed using just thedegrading hash table (e.g., as described in FIG. 37). If a subsequentrequest is a request for data with a new hash not stored in the hashtable array, then the request is processed using the main tree (e.g., asdescribed in FIG. 36), and the array of 1920 hashes 3205 from the newpersist header 3200 that includes the new hash is loaded into the hashtable array.

The hash table array degrades as new arrays of hashes are added becausethe hash table array has a fixed size. For example, the hash table arraycan be configured such that it is approximately 10 times the size of thearray of 1920 hashes 3205. Therefore, once the hash table array fills upwith hashes from various persist headers, as new hashes are added, theprevious hashes are overwritten. This process can be achieved as afunction of adding the hashes to the hash table array without usingother strategies for managing the hash table array (e.g., LRU agingalgorithms).

FIG. 36 is a flowchart depicting the operational flow of a system thatuses scoreboard to find a hash which is not referenced by the scoreboardshown in FIG. 35. Upon a failure, the hash is found in the largerdeduplication indexing system (e.g., shown in FIG. 33) and added to thescoreboard. At step 3601 we lookup data in the scoreboard. At step 3610we test if it is found in the hash table array. At step 3620, if thehash is found, it is returned. At step 3630, if the hash was not foundwe look it up in the main index (e.g., shown in FIG. 33). At step 3640we test if it was found. At step 3650, if the hash is not found wereturn. At step 3660, if step 3640 found the hash in the main index, weretrieve it's persist header (FIG. 32). At step 3670, 3680, and 3690, weiterate through the persist header's “Array of Hashes” and populate thescoreboard with data from the persist header.

FIG. 37 is a flowchart depicting the operational flow of a system thatuses scoreboard to find a hash which is referenced by the scoreboard. Atstep 3701 the scoreboard receives the hash code. At step 3705 a smallportion of the hash code is used to create a “hash fragment.” At step3710 the “hash fragment” is used as an index in to the scoreboard “HashTable Array” (FIG. 35). At step 3715 we test if the “hash fragment”matches the fragment within the “Hash Table Array” entry (FIG. 35). Atstep 3720 we exit the routine with “not found” if the hash does notmatch. At step 3725 we use the “Page Index” from the “Hash Table Array”entry to retrieve the cache page reference of the persist header (FIG.32) which should be in the “Page Cache” (FIG. 34). At step 3730 we testif the correct persist header page could be found in the page cache. Instep 3735 we exit with “not found” if it is not. At step 3740 we comparehash code received in step 3701 with the entry in the persist header. Atstep 3745 we test the result of the comparison. At step 3750 we exitwith “not found” if the hash values do not match. At step 3755 returnwith “found.”

The systems and methods described herein can speed up locating data byusing data locality to augment a generalized index system. Cryptographichash codes such as SHA1, by design, do not provide a way of predictingsubsequent hash values. The scoreboard described herein is a process ofusing the characteristics of data locality to find data without going tothe main hash index (FIG. 33). Further, the degenerating nature of thescoreboard, where there is no active maintenance of the system, reducesthe overhead of managing a more conventional cache system.

System Implementation

FIG. 38 is a diagram that depicts the various components of acomputerized system upon which certain elements may be implemented,according to certain embodiments of the disclosure. The logical modulesdescribed may be implemented on a host computer 3801 that containsvolatile memory 3802, a persistent storage device such as a hard drive,3808, a processor, 3803, and a network interface, 3804. Using thenetwork interface, the system computer can interact with storage pools3805, 3806 over a SAN or Fibre Channel device, among other embodiments.Although FIG. 38 illustrates a system in which the system computer isseparate from the various storage pools, some or all of the storagepools may be housed within the host computer, eliminating the need for anetwork interface. The programmatic processes may be executed on asingle host, as shown in FIG. 38, or they may be distributed acrossmultiple hosts.

The host computer shown in FIG. 38 may serve as an administrativeworkstation, or may implement the application and Application SpecificAgent 402, or may implement any and all logical modules described inthis specification, including the Data Virtualization System itself, ormay serve as a storage controller for exposing storage pools of physicalmedia to the system. Workstations may be connected to a graphicaldisplay device, 3807, and to input devices such as a mouse 3809 and akeyboard 3810. Alternately, the active user's workstation may include ahandheld device.

Throughout this specification reference is made to software components,but all references to software components are intended to apply tosoftware running on hardware. Likewise, objects and data structuresreferred to in the specification are intended to apply to datastructures actually stored in memory, either volatile or non-volatile.Likewise, servers are intended to apply to software, and engines areintended to apply to software, all running on hardware such as thecomputer systems described in FIG. 38.

The foregoing has outlined some of the more pertinent features of thesubject matter. These features should be construed to be merelyillustrative. Many other beneficial results can be attained by applyingthe disclosed subject matter in a different manner or by modifying thesubject matter as will be described.

The invention claimed is:
 1. A method for synchronizing operational datarecords generated during data management operations in a manner thatreduces redundant copying of data during synchronization, comprising:storing, by a host computer, a set of operational data records, whereineach data record is assigned a unique identifier (ID) in ascending orderbased on a creation time of the data record, the host computer includinga processor, the processor coupled to a non-transitory readable mediumhaving instructions executable by the processor; and performing, by thehost computer, a first synchronization operation for the set ofoperational data records, wherein performing the first synchronizationoperation comprises: identifying, by the host computer, a highest uniqueID from the operational data records in the set of operational datarecords associated with a prior synchronization operation, the priorsynchronization operation occurring prior to the first synchronizationoperation; identifying, by the host computer, a tolerance number that isindicative of a range of unique IDs that can be processed at a same timesuch that it cannot be guaranteed that operational data records withunique IDs separated by less than the tolerance number were assignedunique IDs in the order that the operational data records were created,such that the operational data records within the tolerance number maynot have been transferred during the prior synchronization operation;calculating, by the host computer, a synchronization ID comprisingsubtracting the tolerance number from the highest unique ID; andtransmitting, by the host computer, the synchronization ID to a sourceoperational data store to instruct the source operational data store totransmit any operational data records stored at the source operationaldata store with unique IDs greater than the synchronization ID.
 2. Themethod of claim 1, further comprising: identifying a set of unique IDsfrom the set of operational data records, wherein each unique ID in theset of unique IDs is between the highest unique ID and thesynchronization ID; and transmitting the set of unique IDs to the sourceoperational data store to instruct the source operational data store tonot transmit operational data records with the set of unique IDs.
 3. Themethod of claim 1, further comprising transmitting the highest unique IDto the source operational data store to instruct the source operationaldata store to not transmit operational data records with the highestunique ID.
 4. The method of claim 1, further comprising calculating thetolerance number based on a number of concurrent transactions allowedand a maximum delay.
 5. The method of claim 4, further comprisingdoubling the number of concurrent transactions.
 6. The method of claim1, wherein each unique ID is a monotonically-increasing unique ID. 7.The method of claim 1, wherein the set of operational data records is alarge set of operational data records, comprising over 100,000operational data records; and the data records in the set of operationaldata records are not changing once created.
 8. A method forsynchronizing operational data records generated during data managementoperations in a manner that reduces redundant copying of data duringsynchronization, comprising: storing, by a host computer, a set ofoperational data records, wherein each data record is assigned atimestamp based on either a creation time, or modification time of thedata record, the host computer including a processor, the processorcoupled to a non-transitory readable medium having instructionsexecutable by the processor; identifying, by the host computer, a firstbackup time associated with a first synchronization operation of the setof operational data records from a source operational data store; andperforming, by the host computer, a subsequent synchronization operationfor the set of operational data records, the subsequent synchronizationoperation associated with a subsequent backup time, the subsequentbackup time occurring after the first backup time, wherein performingthe subsequent synchronization operation comprises: calculating, by thehost computer, a tolerance number based on an elapsed time that isindicative of a range of timestamps that can be processed at a same timesuch that it cannot be guaranteed that operational data records withtimestamps separated by less than the tolerance number were assignedtimestamps in the order that the operational data records were created,modified, or both, such that the operational data records within thetolerance number may not have been transferred at the first backup time;calculating, by the host computer, a synchronization timestampcomprising subtracting the tolerance number from the first backup time;and transmitting, by the host computer, the synchronization timestamp tothe source operational data store to instruct the source operationaldata store to transmit any operational data records stored at the sourceoperational data store with timestamps greater than the synchronizationtimestamp.
 9. The method of claim 8, further comprising receiving areply comprising: a set of operational data records, each with atimestamp occurring after the synchronization timestamp; and a list ofunique IDs for each data record stored by the source operational datastore.
 10. The method of claim 9, further comprising deleting anyoperational data records in the set of operational data records with aunique ID that is not in the list of unique IDs.
 11. The method of claim8, wherein the set of operational data records is a medium set ofoperational data records, comprising more than 1,000 operational datarecords but less than 100,000 operational data records.
 12. Anon-transitory computer readable medium having executable instructionsoperable to cause an apparatus to: store a set of operational datarecords, wherein each data record is assigned a timestamp based oneither a creation time, or modification time of the data record;identify a first backup time associated with a first synchronizationoperation of the set of operational data records from a sourceoperational data store; and perform a subsequent synchronizationoperation for the set of operational data records, the subsequentsynchronization operation associated with a subsequent backup time, thesubsequent backup time occurring after the first backup time, wherein toperform the subsequent synchronization operation, the apparatus isfurther caused to: calculate a tolerance number based on an elapsed timethat is indicative of a range of timestamps that can be processed at asame time such that it cannot be guaranteed that operational datarecords with timestamps separated by less than the tolerance number wereassigned timestamps in the order that the operational data records werecreated, modified, or both, such that the operational data recordswithin the tolerance number may not have been transferred at the firstbackup time; calculate a synchronization timestamp comprisingsubtracting the tolerance number from the first backup time; andtransmit the synchronization timestamp to the source operational datastore to instruct the source operational data store to transmit anyoperational data records stored at the source operational data storewith timestamps greater than the synchronization timestamp.
 13. Thenon-transitory computer readable medium of claim 12, wherein theapparatus is further caused to receive a reply comprising: a set ofoperational data records, each with a timestamp occurring after thesynchronization timestamp; and a list of unique IDs for each data recordstored by the source operational data store.
 14. The non-transitorycomputer readable medium of claim 13, wherein the apparatus is furthercaused to delete any operational data records in the set of operationaldata records with a unique ID that is not in the list of unique IDs. 15.The non-transitory computer readable medium of claim 12, wherein the setof operational data records is a medium set of operational data records,comprising more than 1,000 operational data records but less than100,000 operational data records.